These 42 commits are when the Protocol Buffers files have changed:
| Commit: | 698d6c3 | |
|---|---|---|
| Author: | Taku Kudo | |
| Committer: | Taku Kudo | |
feat: migrate SentencePiece runtime to sentencepiece_lite (0.3.0) Replace the inference runtime of SentencePieceProcessor with the sentencepiece::lite engine and bump the version to 0.3.0. Runtime: - Remove the legacy models (unigram_model, bpe_model, char_model, word_model, model_interface, model_factory, freelist) and their tests. - Add ModelProtoManager, which validates ModelProto and FlatBuffer inputs, compiles ModelProto to the FlatBuffer model, and rebuilds ModelProto on demand. - Route Encode, SampleEncode, NBestEncode, Decode(ids), Normalize, and the vocabulary accessors through SentencePieceLiteProcessor. Decode(pieces) and WORD model encoding stay in the processor. - Add LoadFromSerializedFlatbuffer(), LoadFromFlatbufferView(), and serialized_flatbuffer(); expose them in Python together with ConvertToFlatbuffer, and drop the memoryview reference when pickling. - normalizer::Normalizer delegates to the lite engine. - lite: keep the bos whitespace state after an <unk> piece that emits an empty unk_surface. API and behavior changes from 0.2.3: - IdToPiece() returns absl::string_view. - Remove SetModel(), SetNormalizer(), and SetNBestTimeout(). - NBestEncode() (Unigram only) no longer builds an A* lattice. The 1st candidate is exact; the others are approximate. - SampleEncode() ignores nbest_size and samples from the full lattice. - status() returns FAILED_PRECONDITION before a model is loaded. - ModelProto.denormalizer_spec is deprecated and ignored. - BPE scores are used as int32 merge ranks, pieces are limited to 4096 bytes, and CHAR models must have single-character NORMAL pieces across both ModelProto and FlatBuffer loading paths. Trainer: - L1 sparse pruning is the only Unigram training algorithm; remove discrete pruning and --use_sparse_pruning. - Scale seed_sentencepiece_size=0 consistently in both corpus and seed-file loading paths. - Build the meta-piece normalizer only when meta pieces exist. - Remove unused helpers and build dependencies. Tests and docs: - Add model_proto_manager_test and util_test cases, port the scenarios of the removed runtime tests to the public API, and restore the trainer Lattice tests. - Add doc/release_notes_0.3.0.md and update cpp.md, options.md, auto_character_coverage.md, and sparse_l1_unigram.md. Known issue: the Protobuf-to-FlatBuffer conversion drops an explicitly empty unk_surface; the corresponding decode test cases (DISABLED_DecodeLeadingUnkWithEmptySurface and the empty unk_surface case of DecodeIdsMatchesDecodePiecesTest) are disabled with TODOs while the fix is on hold.
| Commit: | 2f49713 | |
|---|---|---|
| Author: | Taku Kudo | |
| Committer: | Taku Kudo | |
feat: migrate SentencePiece runtime to sentencepiece_lite (0.3.0) Replace the inference runtime of SentencePieceProcessor with the sentencepiece::lite engine and bump the version to 0.3.0. Runtime: - Remove the legacy models (unigram_model, bpe_model, char_model, word_model, model_interface, model_factory, freelist) and their tests. - Add ModelProtoManager, which validates ModelProto and FlatBuffer inputs, compiles ModelProto to the FlatBuffer model, and rebuilds ModelProto on demand. - Route Encode, SampleEncode, NBestEncode, Decode(ids), Normalize, and the vocabulary accessors through SentencePieceLiteProcessor. Decode(pieces) and WORD model encoding stay in the processor. - Add LoadFromSerializedFlatbuffer(), LoadFromFlatbufferView(), and serialized_flatbuffer(); expose them in Python together with ConvertToFlatbuffer, and drop the memoryview reference when pickling. - normalizer::Normalizer delegates to the lite engine. - lite: keep the bos whitespace state after an <unk> piece that emits an empty unk_surface. API and behavior changes from 0.2.3: - IdToPiece() returns absl::string_view. - Remove SetModel(), SetNormalizer(), and SetNBestTimeout(). - NBestEncode() (Unigram only) no longer builds an A* lattice. The 1st candidate is exact; the others are approximate. - SampleEncode() ignores nbest_size and samples from the full lattice. - status() returns FAILED_PRECONDITION before a model is loaded. - ModelProto.denormalizer_spec is deprecated and ignored. - BPE scores are used as int32 merge ranks, pieces are limited to 4096 bytes, and CHAR models must have single-character NORMAL pieces across both ModelProto and FlatBuffer loading paths. Trainer: - L1 sparse pruning is the only Unigram training algorithm; remove discrete pruning and --use_sparse_pruning. - Scale seed_sentencepiece_size=0 consistently in both corpus and seed-file loading paths. - Build the meta-piece normalizer only when meta pieces exist. - Remove unused helpers and build dependencies. Tests and docs: - Add model_proto_manager_test and util_test cases, port the scenarios of the removed runtime tests to the public API, and restore the trainer Lattice tests. - Add doc/release_notes_0.3.0.md and update cpp.md, options.md, auto_character_coverage.md, and sparse_l1_unigram.md. Known issue: the Protobuf-to-FlatBuffer conversion drops an explicitly empty unk_surface; the corresponding decode test cases (DISABLED_DecodeLeadingUnkWithEmptySurface and the empty unk_surface case of DecodeIdsMatchesDecodePiecesTest) are disabled with TODOs while the fix is on hold.
| Commit: | e269770 | |
|---|---|---|
| Author: | Taku Kudo | |
revert: remove --exclude_pua_codepoints option Equivalent functionality can be achieved via the pretokenizer callback. Following YAGNI, we avoid adding dedicated options with low immediate necessity. This reverts commit bcd4d8c92fa667df2e560fc84291331ce355bf0e.
| Commit: | bcd4d8c | |
|---|---|---|
| Author: | Taku Kudo | |
| Committer: | Taku Kudo | |
feat: add --exclude_pua_codepoints to exclude Unicode PUA from the vocabulary Adds a new TrainerSpec option `exclude_pua_codepoints` (default: false). When enabled, Unicode Private Use Area code points (U+E000..U+F8FF, U+F0000..U+FFFFD, U+100000..U+10FFFD) are excluded from the base character set and from all learned subwords, regardless of their frequency in the training corpus. PUA characters listed in `required_chars` or `user_defined_symbols` are still included. Excluded code points are decomposed into UTF-8 bytes when `byte_fallback=true`, otherwise mapped to <unk>. - proto: new field `exclude_pua_codepoints = 55`; `auto_character_coverage` is renumbered to 56 to keep the field numbering consistent with the internal TrainerSpec (Next id: 57) - spm_train: new `--exclude_pua_codepoints` flag - spec_parser: parse/print support (Python kwargs work automatically) - trainer_interface: filter in LoadSentences() and IsValidSentencePiece() - tests: C++ (trainer_interface_test, sentencepiece_trainer_test) and Python - docs: doc/options.md, doc/special_symbols.md - regenerate python/src/sentencepiece/sentencepiece_model_pb2.py
| Commit: | 70ed5fc | |
|---|---|---|
| Author: | Taku Kudo | |
fix, cleanup, docs: fix empty-sentence removal, remove dead code, and update docs - Fix swap-and-pop index increment when removing empty normalized sentences in TrainerInterface::LoadSentences. - Use InlineVector for the Viterbi DP table in lite::Model::EncodeUnigram and merge duplicate BPE checks in lite::Model::Initialize. - Deduplicate UTF8ToUnicodeOffsets into string_util and remove dead declarations, unused parameters, and redundant build dependencies across src/, lite/, and python/. - Update C++, normalization, CMake, and Lite documentation to match current APIs and benchmarks.
| Commit: | df18339 | |
|---|---|---|
| Author: | Taku Kudo | |
Promote auto_character_coverage to TrainerSpec and remove fixed_sparse_lambda
| Commit: | c2fbf86 | |
|---|---|---|
| Author: | Taku Kudo | |
feat,fix,deprecate: allow unk_id=-1 with byte_fallback, fix issue #1308, and retire deprecated flags - feat(trainer): allow unk_id = -1 when byte_fallback is true - fix(processor): ignore null-byte piece as UNUSED when byte_fallback is enabled (issue #1308) - deprecate: consolidate deprecated parameters in TrainerSpec and retire CLI flags
| Commit: | d29cf00 | |
|---|---|---|
| Author: | Taku Kudo | |
deprecate: consolidate deprecated parameters in TrainerSpec and retire CLI flags
| Commit: | 2f36c31 | |
|---|---|---|
| Author: | Taku Kudo | |
| Committer: | Taku Kudo | |
feat(trainer): allow unk_id = -1 when byte_fallback is true When byte_fallback is enabled, out-of-vocabulary characters are decomposed into byte tokens, making the explicit <unk> token optional. - Allow unk_id = -1 in TrainerInterface and ModelInterface when byte_fallback is true. - Emit a warning when training with unk_id = -1 regarding backward compatibility with SentencePiece <= 0.2.2. - Guard against negative IDs in SentencePieceProcessor (Decode, IsUnknown, IsControl). - Guard against unk_id == -1 in Python bindings when emit_unk_piece is requested. - Update --unk_id flag description and proto documentation. - Add unit tests covering model training, encoding, and decoding with unk_id = -1.
| Commit: | 470b7d5 | |
|---|---|---|
| Author: | Taku Kudo | |
| Committer: | Taku Kudo | |
cleanup: remove self_test_data and consolidate deprecated fields in proto - Remove SelfTestData message and self_test_sample_size/self_test_data from sentencepiece_model.proto. - Consolidate deprecated fields into reserved tags (6, 12, 13, 50, 51, 52 in TrainerSpec; 4 in ModelProto). - Regenerate python/src/sentencepiece/sentencepiece_model_pb2.py. - Remove runtime self-testing loop on model load in sentencepiece_processor.cc. - Remove internal VerifyOutputsEquivalent methods and tests from ModelInterface and UnigramModel. - Remove self_test_sample_size flag and parsing logic from spm_train_main.cc, spec_parser.h, and trainer_interface.cc. - Update third_party/CMakeLists.txt to replace obsolete esaxx with libsais.
| Commit: | 442a7d7 | |
|---|---|---|
| Author: | Taku Kudo | |
deprecate: Clean up deprecated APIs, unused functions, and legacy implementations
| Commit: | db65384 | |
|---|---|---|
| Author: | Taku Kudo | |
fixed minor security issue
The documentation is generated from this commit.
| Commit: | 53de765 | |
|---|---|---|
| Author: | Taku Kudo | |
allows to load precomputed seed sentencepieces for unigram from a file.
| Commit: | e58bb68 | |
|---|---|---|
| Author: | Taku Kudo | |
add pretokenization_delimiter options. Initialize seed pieces more accurately.
| Commit: | 9f3ed99 | |
|---|---|---|
| Author: | Taku Kudo | |
Sync internal to github. DP related features are added.
| Commit: | fab966a | |
|---|---|---|
| Author: | Taku Kudo | |
sync from internal
| Commit: | 3a5bc58 | |
|---|---|---|
| Author: | Taku Kudo | |
Revert "sync from internal" This reverts commit 05db0894d8ea44b203c3501306061cde9e42c48e.
| Commit: | 05db089 | |
|---|---|---|
| Author: | Taku Kudo | |
sync from internal
| Commit: | 8eaa672 | |
|---|---|---|
| Author: | Taku Kudo | |
change the type of input_sentence_size from int32 to uint64
| Commit: | f7bc3db | |
|---|---|---|
| Author: | Taku Kudo | |
merges internal changes to github
| Commit: | 8b921ac | |
|---|---|---|
| Author: | Taku Kudo | |
Revert the default size of piece length.
| Commit: | 329383b | |
|---|---|---|
| Author: | Taku Kudo | |
Initial release of 0.19. Merged internal sentencepiece.
| Commit: | 61d59db | |
|---|---|---|
| Author: | Sentencepiece Team | |
| Committer: | Keith Stevens | |
Project import generated by Copybara. PiperOrigin-RevId: 251772401
| Commit: | 59b48eb | |
|---|---|---|
| Author: | Taku Kudo | |
added --treat_whitespace_as_suffix option to make _ be a suffix of word.
| Commit: | 7b19d68 | |
|---|---|---|
| Author: | Taku Kudo | |
use builtin protobuf-lite package in third_party
| Commit: | 904edfe | |
|---|---|---|
| Author: | Taku Kudo | |
deperecated mining_sentence_size and training_sentence_size. Load all sentences by default.
| Commit: | 5f635d0 | |
|---|---|---|
| Author: | Taku Kudo | |
support to change the piece of unk/bos/eos/pad
| Commit: | 55e4da4 | |
|---|---|---|
| Author: | Taku Kudo | |
added --max_sentence_length flag
| Commit: | 6d0fd75 | |
|---|---|---|
| Author: | Taku Kudo | |
added --split_by_number flag
| Commit: | b40cca7 | |
|---|---|---|
| Author: | Taku Kudo | |
Added --use_all_vocab=true flag for WORD/CHAR model
| Commit: | b6a74ee | |
|---|---|---|
| Author: | Taku Kudo | |
Added self testing feature.
| Commit: | 5dac483 | |
|---|---|---|
| Author: | Taku Kudo | |
Added --unk_surface option to allow user to change unknown surface string.
| Commit: | e437e30 | |
|---|---|---|
| Author: | Taku Kudo | |
Support vocab restriction feature
| Commit: | f228e55 | |
|---|---|---|
| Author: | Taku Kudo | |
Reimplement Trainer with Proto reflection
| Commit: | ca8754b | |
|---|---|---|
| Author: | Taku Kudo | |
Add --hard_vocab_limit flag.
| Commit: | ecbd55a | |
|---|---|---|
| Author: | Taku Kudo | |
| Committer: | GitHub | |
Merge pull request #53 from google/sr Support to change ids of <unk>, <s>, </s>
| Commit: | d102897 | |
|---|---|---|
| Author: | Taku Kudo | |
Support to change ids of <unk>, <s>, </s>
| Commit: | ea41f40 | |
|---|---|---|
| Author: | Taku Kudo | |
| Committer: | GitHub | |
Merge pull request #48 from tetsuok/fix-default-vocab-size Fix inconsistent default vocab size
| Commit: | 9f3bc53 | |
|---|---|---|
| Author: | Tetsuo Kiso | |
Fix typo in examples of control symbols
| Commit: | ca20a5a | |
|---|---|---|
| Author: | Tetsuo Kiso | |
Fix inconsistent default vocab size
| Commit: | c6a1a19 | |
|---|---|---|
| Author: | Taku Kudo | |
Add Sample/NBestEncode
| Commit: | 2928ce5 | |
|---|---|---|
| Author: | Taku Kudo | |
Initialize repository