package zemberek.preprocessor

Get desktop application:
View/edit binary Protocol Buffers messages

• rpc Tokenize (TokenizationRequest, TokenizationResponse)

  preprocess.proto:37

  message TokenizationRequest

  preprocess.proto:8

  • string input = 1

  • bool includeTokenBoundaries = 2

  • bool ignoreDoubleQuotes = 3

    TODO: Add options (ignore etc.)

  message TokenizationResponse

  preprocess.proto:23

  • repeated TokenProto tokens = 1

• rpc ExtractSentences (SentenceExtractionRequest, SentenceExtractionResponse)

  preprocess.proto:38

  message SentenceExtractionRequest

  preprocess.proto:27

  • string document = 1

  • bool doNotSplitInDoubleQuotes = 2

  message SentenceExtractionResponse

  preprocess.proto:32

  • repeated string sentences = 1

Used in: TokenizationResponse

• string token = 1

• string type = 2

  TODO: Types should be enums.

• int32 start = 3

• int32 end = 4