package allspark.allspark_service

Get desktop application:
View/edit binary Protocol Buffers messages

Interface exported by the service.

• rpc BuildModelFromConfigStruct (ModelStructConfig, AsStatus)

  allspark_service.proto:14

  Build model from structure config.

  message ModelStructConfig

  allspark_service.proto:210

  • string model_name = 1

  • string model_path = 2

  • string weights_path = 3

  • string compute_unit = 4

  • string matmul_precision = 5

  • bool is_lora_cfg = 6

  • int64 swap_threshold = 7

  • int32 engine_max_length = 8

  • int32 engine_max_batch = 9

  • optional LoraNames lora_names = 10

  • int32 cache_span_size = 11

  • int32 cache_span_num_init = 12

  • int32 cache_span_num_grow = 13

  • AsCacheMode cache_mode = 14

  • bool enable_prefix_cache = 15

  • int32 prefix_cache_ttl = 16

  • AsMHAPrefill prefill_mode = 17

  • bool text_graph = 18

  • int32 num_threads = 19

• rpc ReloadModelToDeviceMemory (ModelName, AsStatus)

  allspark_service.proto:16

• rpc UnloadModelFromDeviceMemory (ModelName, AsStatus)

  allspark_service.proto:18

• rpc GetModelInformation (ModelName, ModelInfo)

  allspark_service.proto:21

  Get model type, input tensor , output tensor info

  message ModelInfo

  allspark_service.proto:339

  • string model_info = 1

• rpc GetFileInformation (FileInformationRequest, FileInformationResponse)

  allspark_service.proto:23

  message FileInformationRequest

  allspark_service.proto:271

  • string model_path = 1

  • string param_path = 2

  message FileInformationResponse

  allspark_service.proto:276

  • string create_version_graph = 1

  • string create_version_param = 2

  • string current_version_engine = 3

• rpc StartModel (ModelName, AsStatus)

  allspark_service.proto:25

  start model

• rpc StopModel (ModelName, AsStatus)

  allspark_service.proto:28

  stop model

• rpc ReleaseModel (ModelName, AsStatus)

  allspark_service.proto:31

  release model

• rpc StartRequest (StartRequestRequest, StartRequestResponse)

  allspark_service.proto:34

  start request

  message StartRequestRequest

  allspark_service.proto:253

  • string model_name = 1

  • RequestInferType infer_type = 2

  • RequestMMType mm_type = 3

  • optional TensorMap inputs = 4

  • optional GenerateConfig config = 6

• rpc StopRequest (StopRequestRequest, StartRequestResponse)

  allspark_service.proto:37

  stop request

• rpc ReleaseRequest (StopRequestRequest, StartRequestResponse)

  allspark_service.proto:40

  release request

• rpc SyncRequest (StopRequestRequest, StartRequestResponse)

  allspark_service.proto:43

  sync request

• rpc GetAsEngineStat (ModelName, AsEngineStat)

  allspark_service.proto:45

  message AsEngineStat

  allspark_service.proto:232

  • string model_name = 1

  • int64 total_span = 2

  • int64 used_span = 3

  • int64 free_span = 4

  • int64 free_token = 5

  • int64 span_size = 6

  • int32 pendding_request = 7

  • int32 running_request = 8

  • int64 total_device_memory_pool_size = 9

  • int64 used_device_memory_pool_size = 10

  • int64 total_generated_token = 11

  • int64 total_prefill_token = 12

  • float generate_token_persec = 13

  • float process_token_persec = 14

  • int64 prefix_cache_hit_token = 15

  • int64 prefix_cache_miss_token = 16

  • float prefix_cache_hit_rate = 17

  • float prefix_cache_miss_rate = 18

• rpc GetVersionFull (Empty, VersionInfo)

  allspark_service.proto:49

  Get SDK version string version string will include version, git sha1, and build time, eg: 0.1.4/(GitSha1:beaca93)/(Build:20230403154806)

  message VersionInfo

  allspark_service.proto:343

  • string version_info = 1

• rpc GetOpProfilingInfo (ModelName, OpProfilingInfo)

  allspark_service.proto:53

  Get op profiling info profiling string includes op name, min_time, max_time, count, sum, percentage

  message OpProfilingInfo

  allspark_service.proto:347

  • string op_profiling_info = 1

• rpc GetRankId (Empty, RankId)

  allspark_service.proto:61

  Get rank id (0~rank_num-1) Since openmpi is used to manage CPU inferer task, which may launch multiply process to do the inferer, GetRankId is used to indicate the manager process and get the output in manager process. @note 0 is the manager process, we get output only if GetRandId return 0. GetRankId always return 0 in GPU inferer.

• rpc GetRankNums (Empty, RankId)

  allspark_service.proto:65

  Get rank num It is used to check if service is still alive

• rpc ShutdownService (Empty, AsStatus)

  allspark_service.proto:69

  shutdown service

• rpc GenerateStatus (UUID, GenerateRequestStatus)

  allspark_service.proto:72

  callback for ResultQueue

  message GenerateRequestStatus

  allspark_service.proto:183

  • GENERATE_STATUS status = 1

• rpc GeneratedLength (UUID, GenerateLen)

  allspark_service.proto:73

  message GenerateLen

  allspark_service.proto:179

  • int64 len = 1

• rpc Get (UUID, GeneratedElements)

  allspark_service.proto:74

• rpc GetNoWait (UUID, GeneratedElements)

  allspark_service.proto:75

AllSpark status code

Used in: AsStatus, StartRequestResponse

• ALLSPARK_SUCCESS = 0

  正常状态

• ALLSPARK_UNKNOWN_ERROR = 1

  兜底错误类型

• ALLSPARK_PARAM_ERROR = 2

  传入的参数值错误

• ALLSPARK_IO_ERROR = 3

  IO访问错误

• ALLSPARK_MEMORY_ERROR = 4

  内存越界等错误

• ALLSPARK_RUNTIME_ERROR = 5

  运行时触发的OP错误返回

• ALLSPARK_EXCEED_LIMIT_ERROR = 7

  参数或输入超限

• ALLSPARK_INVALID_CALL_ERROR = 8

  无效调用

• ALLSPARK_EMPTY_REQUEST = 9

  无有效请求

• ALLSPARK_ILLEGAL_REQUEST_ID = 10

  没有找到有效的request_id

• ALLSPARK_CACHE_MEMORY_OUT = 11

• ALLSPARK_REQUEST_DENIED = 12

  停服务状态，拒绝服务

• ALLSPARK_STREAMING = 200

  ALLSPARK_LORA_NOT_LOADED = 13; // 用户指定的lora没加载， 已经停用

  流式返回

Used in: ModelStructConfig

• AsCacheDefault = 0

Used in: ModelStructConfig

• AsPrefillDefault = 0

• AsPrefillFlashV2 = 10

Used as response type in: AllSpark.BuildModelFromConfigStruct, AllSpark.ReleaseModel, AllSpark.ReloadModelToDeviceMemory, AllSpark.ShutdownService, AllSpark.StartModel, AllSpark.StopModel, AllSpark.UnloadModelFromDeviceMemory

• AS_STATUS as_status = 1

Used in: GenerateConfig

• repeated BadWordIds.Array ids = 1

Used in: BadWordIds

• repeated int32 word = 1

Used in: Tensor

• DENSE = 0

• CSC = 1

• ELL = 2

Used in: Tensor

• DATATYPE_UNDEFINED = 0

• FLOAT32 = 1

• FLOAT16 = 2

• INT8 = 3

• INT16 = 4

• INT32 = 5

• INT64 = 6

• STRING = 7

• BOOL = 8

• BFLOAT16 = 9

• UINT8 = 10

Used in: DeviceType

• DEVICETYPE_UNDEFINED = 0

• CPU = 1

• CUDA = 2

Used in: Tensor

• DEVICE_TYPE dev_type = 1

Used as request type in: AllSpark.GetRankId, AllSpark.GetRankNums, AllSpark.GetVersionFull, AllSpark.ShutdownService

(message has no fields)

Used in: GenerateRequestStatus

• Init = 0

  / Init status.

• ContextFinished = 1

  / Context computation finished.

• Generating = 2

  / Start generating.

• GenerateFinished = 3

  / Generation finished, EOS token was generated.

• GenerateInterrupted = 4

  / The Generation was interrupted, often means

Used in: StartRequestRequest

• optional BadWordIds bad_words_ids = 1

• optional StopWordIds stop_words_ids = 2

• bool do_sample = 3

• bool early_stopping = 4

• bool async = 5

• int32 async_token_num = 6

• int32 first_token_num = 7

• string uuid = 8

• int32 num_beams = 9

• int32 num_return_sequences = 10

• float temperature = 11

• float top_p = 12

• float repetition_penalty = 13

• float length_penalty = 14

• float presence_penalty = 15

• bool suppress_repetition_in_generation = 16

• int32 min_length = 17

• int32 max_length = 18

• int32 no_repeat_ngram_size = 19

• int32 eos_token_id = 20

• int32 top_k = 21

• int32 input_len = 22

• int64 seed = 23

• bool loop_context = 24

• repeated int32 top_k_arr = 25

• repeated float top_p_arr = 26

• repeated int64 seed_arr = 27

• bool probe_only = 28

• string lora_name = 29

• map<string, string> response_format = 30

• map<string, int32> vocab = 31

• VocabType vocab_type = 32

Used as response type in: AllSpark.Get, AllSpark.GetNoWait

• int32 empty = 1

• repeated int64 ids_from_generate = 2

• optional TensorMap tensors_from_model_inference = 3

Used in: ModelStructConfig

• repeated string names = 1

Used as request type in: AllSpark.GetAsEngineStat, AllSpark.GetModelInformation, AllSpark.GetOpProfilingInfo, AllSpark.ReleaseModel, AllSpark.ReloadModelToDeviceMemory, AllSpark.StartModel, AllSpark.StopModel, AllSpark.UnloadModelFromDeviceMemory

• string model_name = 1

Used as response type in: AllSpark.GetRankId, AllSpark.GetRankNums

• int32 rank_id = 1

Used in: StartRequestRequest

• Generate = 0

• ModelInference = 1

Used in: StartRequestRequest

• TextInput = 0

• MultiMediaTypeRichText = 1

Used as response type in: AllSpark.ReleaseRequest, AllSpark.StartRequest, AllSpark.StopRequest, AllSpark.SyncRequest

• AS_STATUS as_status = 1

• string uuid = 2

Used as request type in: AllSpark.ReleaseRequest, AllSpark.StopRequest, AllSpark.SyncRequest

• string model_name = 1

• string uuid = 2

Used in: GenerateConfig

• repeated StopWordIds.Array ids = 1

Used in: StopWordIds

• repeated int64 word = 1

Used in: TensorListMap.Array, TensorMap

• string name = 1

• optional Tensor.Shape shape = 2

• optional DeviceType device_type = 3

• DATA_TYPE data_type = 4

• DATA_MODE data_mode = 5

• bytes data = 6

Used in: Tensor

• repeated int32 dims = 1

• map<string, TensorListMap.Array> tensor_list_map = 1

Used in: TensorListMap

• repeated Tensor tensor = 1

Used in: GeneratedElements, StartRequestRequest

• map<string, Tensor> tensor_map = 1

Used as request type in: AllSpark.GenerateStatus, AllSpark.GeneratedLength, AllSpark.Get, AllSpark.GetNoWait

• string uuid = 1

Used in: GenerateConfig

• VOCAB_TYPE_WPM = 0

• VOCAB_TYPE_SPM = 1

• VOCAB_TYPE_UGM = 2

• VOCAB_TYPE_BPE = 3