package sax.server.lm

Get desktop application:
View/edit binary Protocol Buffers messages

• rpc Score (ScoreRequest, ScoreResponse)

  lm.proto:140

  Returns the score (e.g., log pplx) given the text.

  message ScoreRequest

  lm.proto:39

  • string model_key = 1

  • repeated string suffix = 2

  • string prefix = 3

    If `prefix` is not empty, it will be prepended to `suffix`, but the score will be calculated only on `suffix`.

  • optional ExtraInputs extra_inputs = 4

  message ScoreResponse

  lm.proto:48

  • repeated double logp = 1

• rpc Generate (GenerateRequest, GenerateResponse)

  lm.proto:143

  Returns generated texts using the text prefix in the request.

  message GenerateResponse

  lm.proto:67

  • repeated DecodedText texts = 1

• rpc GenerateStream (GenerateRequest, stream GenerateStreamResponse)

  lm.proto:146

  Returns a stream of generated texts using the text prefix in the request.

• rpc Embed (EmbedRequest, EmbedResponse)

  lm.proto:149

  Returns a text embedding given the text.

  message EmbedRequest

  lm.proto:110

  • string model_key = 1

  • string text = 2

  • optional ExtraInputs extra_inputs = 3

  message EmbedResponse

  lm.proto:116

  • repeated double embedding = 1

• rpc Gradient (GradientRequest, GradientResponse)

  lm.proto:152

  Returns the score and gradients given the text.

  message GradientRequest

  lm.proto:120

  • string model_key = 1

  • string suffix = 2

  • string prefix = 3

  • optional ExtraInputs extra_inputs = 4

  message GradientResponse

  lm.proto:127

  • repeated double score = 1

  • map<string, GradientResponse.Gradient> gradients = 2

    Map of gradients of tensors keyed by tensor name.

Used in: GenerateResponse

• string text = 1

  The decoded text. Depending on the servable model params, the result may or may not include the input prefix text in SamplingDecodeRequest. See DecodeHParams in servable_lm_model_param.py.

• double score = 2

Used as request type in: LMService.Generate, LMService.GenerateStream

• string model_key = 1

• string text = 2

  The prefix text.

• optional ExtraInputs extra_inputs = 3

Usage example: results = [""] * num_of_samples scores = [0.0] * num_of_samples for response in range(get_streaming_responses()): for i, item in enumerate(response.items): results[i] = results[i][:item.prefix_len] + item.text scores[i] = item.score[0] results[i] and scores[i] will contain the fully decoded text and score for the ith item.

Used in: GenerateStreamResponse

• string text = 1

  The decoded text. This contains new, incrementally decoded text not included in the result accumulated so far.

• int32 prefix_len = 2

  How many leading characters in the result accumulated so far should prefix the `text` field above.

• double score = 3

  The final response in a stream is required to contain scores for each fully decoded item. TODO(b/286079331): Remove this field.

• repeated double scores = 4

  Various scores corresponding to each stream item (e.g. log_prob or scores for attributes such safety, quality, etc.).

Each response message represents one incremental decoding result in a streaming of results.

Used as response type in: LMService.GenerateStream, vertex.PredictionService.PredictStreamed

• repeated GenerateStreamItem items = 1

  Each item corresponds to one of several possible decoded suffixes. The server decides on the order of these items. In other words, the order is not guaranteed to be stable between responses.

Used in: GradientResponse

• repeated double values = 1