package generate.v2

Get desktop application:
View/edit binary Protocol Buffers messages

• rpc Info (InfoRequest, InfoResponse)

  generate.proto:7

  / Model Info

  message InfoRequest

  generate.proto:28

  / Empty request

  (message has no fields)

  message InfoResponse

  generate.proto:30

  • bool requires_padding = 1

  • string dtype = 2

  • string device_type = 3

  • optional uint32 window_size = 4

  • uint32 speculate = 5

• rpc ServiceDiscovery (ServiceDiscoveryRequest, ServiceDiscoveryResponse)

  generate.proto:9

  / Service discovery

  message ServiceDiscoveryRequest

  generate.proto:39

  / Empty request

  (message has no fields)

  message ServiceDiscoveryResponse

  generate.proto:41

  • repeated string urls = 1

    / Other shards urls

• rpc ClearCache (ClearCacheRequest, ClearCacheResponse)

  generate.proto:11

  / Empties batch cache

  message ClearCacheRequest

  generate.proto:46

  • optional uint64 id = 1

    / Optional batch id

  message ClearCacheResponse

  generate.proto:52

  / Empty response

  (message has no fields)

• rpc FilterBatch (FilterBatchRequest, FilterBatchResponse)

  generate.proto:13

  / Remove requests from a cached batch

  message FilterBatchRequest

  generate.proto:174

  • uint64 batch_id = 1

    / Batch ID

  • repeated uint64 request_ids = 2

    / Requests to keep

  message FilterBatchResponse

  generate.proto:181

  • optional CachedBatch batch = 1

    / Filtered Batch (cached)

• rpc Warmup (WarmupRequest, WarmupResponse)

  generate.proto:15

  / Warmup the model and compute max cache size

  message WarmupRequest

  generate.proto:225

  • optional Batch batch = 1

    / Batch to warmup on

  • uint32 max_input_length = 2

  • uint32 max_prefill_tokens = 3

  • uint32 max_total_tokens = 4

  message WarmupResponse

  generate.proto:233

  • optional uint32 max_supported_total_tokens = 1

    / Maximum number of tokens supported by the model

• rpc Prefill (PrefillRequest, PrefillResponse)

  generate.proto:17

  / Prefill batch and decode first token

  message PrefillRequest

  generate.proto:187

  • optional Batch batch = 1

    / Batch

  message PrefillResponse

  generate.proto:192

  • repeated Generation generations = 1

    / Generation

  • optional CachedBatch batch = 2

    / Next batch (cached)

  • uint64 forward_ns = 3

    / Forward elapsed time in nanoseconds

  • uint64 decode_ns = 4

    / Decode elapsed time in nanoseconds

  • uint64 total_ns = 5

    / Total elapsed time in nanoseconds

• rpc Decode (DecodeRequest, DecodeResponse)

  generate.proto:19

  / Decode token for a list of prefilled batches

  message DecodeRequest

  generate.proto:205

  • repeated CachedBatch batches = 1

    / Cached batches

  message DecodeResponse

  generate.proto:210

  • repeated Generation generations = 1

    / Decodes

  • optional CachedBatch batch = 2

    / Next batch (cached)

  • uint64 forward_ns = 3

    / Forward elapsed time in nanoseconds

  • uint64 decode_ns = 4

    / Decode elapsed time in nanoseconds

  • uint64 total_ns = 5

    / Total elapsed time in nanoseconds

  • optional uint64 concat_ns = 6

    / Concatenate elapsed time in nanoseconds

• rpc Health (HealthRequest, HealthResponse)

  generate.proto:21

  / Health check

  message HealthRequest

  generate.proto:24

  (message has no fields)

  message HealthResponse

  generate.proto:25

  (message has no fields)

Used in: PrefillRequest, WarmupRequest

• uint64 id = 1

  / Batch ID

• repeated Request requests = 2

  / Individual requests

• uint32 size = 3

  / Batch size (==len(requests))

• uint32 max_tokens = 4

  / Maximum number of tokens this batch will grow to

Used in: DecodeRequest, DecodeResponse, FilterBatchResponse, PrefillResponse

• uint64 id = 1

  / Batch ID

• repeated uint64 request_ids = 2

  / Individual requests ids

• uint32 size = 3

  / Batch size (==len(requests))

• uint32 max_tokens = 4

  / Maximum number of tokens this batch will grow to

Used in: GeneratedText

• FINISH_REASON_LENGTH = 0

• FINISH_REASON_EOS_TOKEN = 1

• FINISH_REASON_STOP_SEQUENCE = 2

Used in: Generation

• string text = 1

  / Output

• uint32 generated_tokens = 2

  / Number of generated tokens

• FinishReason finish_reason = 3

  / Finish reason

• optional uint64 seed = 4

  / Seed

Used in: DecodeResponse, PrefillResponse

• uint64 request_id = 1

  / Request ID

• optional Tokens prefill_tokens = 2

  / Prefill tokens (optional)

• optional Tokens tokens = 3

• optional GeneratedText generated_text = 4

  / Complete generated text

• repeated Tokens top_tokens = 5

  / Top tokens

Used in: NextTokenChooserParameters

• GRAMMAR_TYPE_NONE = 0

• GRAMMAR_TYPE_JSON = 1

• GRAMMAR_TYPE_REGEX = 2

Used in: Request

• float temperature = 1

  / exponential scaling output probability distribution

• uint32 top_k = 2

  / restricting to the k highest probability elements

• float top_p = 3

  / restricting to top tokens summing to prob_cut_off <= prob_cut_off

• float typical_p = 4

  / restricting to top tokens summing to prob_cut_off <= prob_cut_off

• bool do_sample = 5

  / apply sampling on the logits

• uint64 seed = 6

  / random seed for sampling

• float repetition_penalty = 7

  / repetition penalty

• float frequency_penalty = 9

  / frequency penalty

• bool watermark = 8

  / token watermarking using "A Watermark for Large Language Models"

• string grammar = 10

  / grammar (applied if not empty)

• GrammarType grammar_type = 11

  / grammar type

Used in: Batch

• uint64 id = 1

  / Request ID

• string inputs = 2

  / The generation context

• uint32 truncate = 3

  / Context truncation

• optional NextTokenChooserParameters parameters = 4

  / Next Token Chooser Parameters

• optional StoppingCriteriaParameters stopping_parameters = 5

  / Stopping Criteria Parameters

• bool prefill_logprobs = 6

  / Return prefill logprobs

• uint32 top_n_tokens = 7

  / Return most likely n tokens

Used in: Request

• uint32 max_new_tokens = 1

  / Maximum number of generated tokens

• repeated string stop_sequences = 2

  / Optional stopping sequences

• bool ignore_eos_token = 3

  / Ignore end of sequence token / used for benchmarking

Used in: Generation

• repeated uint32 ids = 1

  / Token IDs

• repeated float logprobs = 2

  / Logprobs

• repeated string texts = 3

  / tokens

• repeated bool is_special = 4

  / special