package sax.server.multimodal

Get desktop application:
View/edit binary Protocol Buffers messages

• rpc Embed (EmbedRpcRequest, EmbedRpcResponse)

  multimodal.proto:141

  Returns an embedding given the data items.

  message EmbedRpcRequest

  multimodal.proto:117

  • string model_key = 1

  • optional EmbedRequest request = 2

  • optional ExtraInputs extra_inputs = 3

  message EmbedRpcResponse

  multimodal.proto:131

  • optional EmbedResponse response = 1

• rpc Generate (GenerateRpcRequest, GenerateRpcResponse)

  multimodal.proto:137

  Returns generated results using the data items in the request.

  message GenerateRpcRequest

  multimodal.proto:111

  • string model_key = 1

  • optional GenerateRequest request = 2

  • optional ExtraInputs extra_inputs = 3

  message GenerateRpcResponse

  multimodal.proto:127

  • optional GenerateResponse response = 1

• rpc Score (ScoreRpcRequest, ScoreRpcResponse)

  multimodal.proto:139

  Returns score results of the suffix items in the request.

  message ScoreRpcRequest

  multimodal.proto:105

  • string model_key = 1

  • optional ScoreRequest request = 2

  • optional ExtraInputs extra_inputs = 3

  message ScoreRpcResponse

  multimodal.proto:123

  • optional ScoreResponse response = 1

Used in: DataItem

• repeated float sample = 1

Used in: EmbedRequest, GenerateRequest, GenerateResult, ScoreRequest

• oneof item

  • string text = 1

  • bytes image_bytes = 2

  • IdList image_tokens = 3

  • FloatEmbedding image_embedding = 6

  • AudioFloatSamples audio_float_samples = 4

  • IdList audio_tokens = 7

  • bytes video = 5

    'video' is up to the specific model to interpret. E.g., it can be the content of a mpeg4 file.

  • MediaPath media_path = 8

Used in: EmbedResponse

• repeated double values = 1

Used in: EmbedRpcRequest

• repeated DataItem items = 1

Used in: EmbedRpcResponse

• optional DoubleEmbedding embedding = 1

Used in: DataItem

• repeated float values = 1

Used in: GenerateRpcRequest

• repeated DataItem items = 1

Used in: GenerateRpcResponse

• repeated GenerateResult results = 1

  len(result) == # samples

Used in: GenerateResponse

• repeated DataItem items = 1

• double score = 2

Used in: DataItem

• repeated int64 id = 1

Used in: DataItem

• string media_path = 1

• MediaPath.MediaType media_type = 3

Used in: MediaPath

• UNSPECIFIED = 0

• IMAGE = 1

• AUDIO = 2

• VIDEO = 3

Used in: ScoreRpcRequest

• repeated DataItem prefix_items = 1

• repeated DataItem suffix_items = 2

Used in: ScoreRpcResponse

• repeated ScoreResult results = 1

Used in: ScoreResponse

• double score = 1