package milvus.proto.tokenizer

Get desktop application:
View/edit binary Protocol Buffers messages

• rpc Tokenize (TokenizationRequest, TokenizationResponse)

  tokenizer.proto:6

  message TokenizationRequest

  tokenizer.proto:9

  • string text = 1

  • repeated TokenizationRequest.Parameter parameters = 2

  message TokenizationResponse

  tokenizer.proto:18

  • repeated Token tokens = 1

Used in: TokenizationResponse

• string text = 1

• int32 offset_from = 2

• int32 offset_to = 3

• int32 position = 4

• int32 position_length = 5

Used in: TokenizationRequest

• string key = 1

• repeated string values = 2