protodoc.io
Project AI-Hypercomputer/JetStream
Welcome
Readme
License
Commits
GitHub project
Download proto files
Desktop application
Contact us
Default package
engine
jetstream_proto
engine
TokenizerParameters
TokenizerType
package
engine
Get desktop application:
View/edit binary Protocol Buffers messages
message
TokenizerParameters
tokenizer.proto:21
string
path
= 1
TokenizerType
tokenizer_type
= 2
string
access_token
= 3
bool
use_chat_template
= 4
int32
extra_ids
= 5
enum
TokenizerType
tokenizer.proto:29
Used in:
TokenizerParameters
tiktoken
= 0
sentencepiece
= 1
huggingface
= 2