Proto commits in google/sentencepiece

These 42 commits are when the Protocol Buffers files have changed:

Commit:698d6c3
Author:Taku Kudo
Committer:Taku Kudo

feat: migrate SentencePiece runtime to sentencepiece_lite (0.3.0) Replace the inference runtime of SentencePieceProcessor with the sentencepiece::lite engine and bump the version to 0.3.0. Runtime: - Remove the legacy models (unigram_model, bpe_model, char_model, word_model, model_interface, model_factory, freelist) and their tests. - Add ModelProtoManager, which validates ModelProto and FlatBuffer inputs, compiles ModelProto to the FlatBuffer model, and rebuilds ModelProto on demand. - Route Encode, SampleEncode, NBestEncode, Decode(ids), Normalize, and the vocabulary accessors through SentencePieceLiteProcessor. Decode(pieces) and WORD model encoding stay in the processor. - Add LoadFromSerializedFlatbuffer(), LoadFromFlatbufferView(), and serialized_flatbuffer(); expose them in Python together with ConvertToFlatbuffer, and drop the memoryview reference when pickling. - normalizer::Normalizer delegates to the lite engine. - lite: keep the bos whitespace state after an <unk> piece that emits an empty unk_surface. API and behavior changes from 0.2.3: - IdToPiece() returns absl::string_view. - Remove SetModel(), SetNormalizer(), and SetNBestTimeout(). - NBestEncode() (Unigram only) no longer builds an A* lattice. The 1st candidate is exact; the others are approximate. - SampleEncode() ignores nbest_size and samples from the full lattice. - status() returns FAILED_PRECONDITION before a model is loaded. - ModelProto.denormalizer_spec is deprecated and ignored. - BPE scores are used as int32 merge ranks, pieces are limited to 4096 bytes, and CHAR models must have single-character NORMAL pieces across both ModelProto and FlatBuffer loading paths. Trainer: - L1 sparse pruning is the only Unigram training algorithm; remove discrete pruning and --use_sparse_pruning. - Scale seed_sentencepiece_size=0 consistently in both corpus and seed-file loading paths. - Build the meta-piece normalizer only when meta pieces exist. - Remove unused helpers and build dependencies. Tests and docs: - Add model_proto_manager_test and util_test cases, port the scenarios of the removed runtime tests to the public API, and restore the trainer Lattice tests. - Add doc/release_notes_0.3.0.md and update cpp.md, options.md, auto_character_coverage.md, and sparse_l1_unigram.md. Known issue: the Protobuf-to-FlatBuffer conversion drops an explicitly empty unk_surface; the corresponding decode test cases (DISABLED_DecodeLeadingUnkWithEmptySurface and the empty unk_surface case of DecodeIdsMatchesDecodePiecesTest) are disabled with TODOs while the fix is on hold.

Commit:2f49713
Author:Taku Kudo
Committer:Taku Kudo

feat: migrate SentencePiece runtime to sentencepiece_lite (0.3.0) Replace the inference runtime of SentencePieceProcessor with the sentencepiece::lite engine and bump the version to 0.3.0. Runtime: - Remove the legacy models (unigram_model, bpe_model, char_model, word_model, model_interface, model_factory, freelist) and their tests. - Add ModelProtoManager, which validates ModelProto and FlatBuffer inputs, compiles ModelProto to the FlatBuffer model, and rebuilds ModelProto on demand. - Route Encode, SampleEncode, NBestEncode, Decode(ids), Normalize, and the vocabulary accessors through SentencePieceLiteProcessor. Decode(pieces) and WORD model encoding stay in the processor. - Add LoadFromSerializedFlatbuffer(), LoadFromFlatbufferView(), and serialized_flatbuffer(); expose them in Python together with ConvertToFlatbuffer, and drop the memoryview reference when pickling. - normalizer::Normalizer delegates to the lite engine. - lite: keep the bos whitespace state after an <unk> piece that emits an empty unk_surface. API and behavior changes from 0.2.3: - IdToPiece() returns absl::string_view. - Remove SetModel(), SetNormalizer(), and SetNBestTimeout(). - NBestEncode() (Unigram only) no longer builds an A* lattice. The 1st candidate is exact; the others are approximate. - SampleEncode() ignores nbest_size and samples from the full lattice. - status() returns FAILED_PRECONDITION before a model is loaded. - ModelProto.denormalizer_spec is deprecated and ignored. - BPE scores are used as int32 merge ranks, pieces are limited to 4096 bytes, and CHAR models must have single-character NORMAL pieces across both ModelProto and FlatBuffer loading paths. Trainer: - L1 sparse pruning is the only Unigram training algorithm; remove discrete pruning and --use_sparse_pruning. - Scale seed_sentencepiece_size=0 consistently in both corpus and seed-file loading paths. - Build the meta-piece normalizer only when meta pieces exist. - Remove unused helpers and build dependencies. Tests and docs: - Add model_proto_manager_test and util_test cases, port the scenarios of the removed runtime tests to the public API, and restore the trainer Lattice tests. - Add doc/release_notes_0.3.0.md and update cpp.md, options.md, auto_character_coverage.md, and sparse_l1_unigram.md. Known issue: the Protobuf-to-FlatBuffer conversion drops an explicitly empty unk_surface; the corresponding decode test cases (DISABLED_DecodeLeadingUnkWithEmptySurface and the empty unk_surface case of DecodeIdsMatchesDecodePiecesTest) are disabled with TODOs while the fix is on hold.

Commit:e269770
Author:Taku Kudo

revert: remove --exclude_pua_codepoints option Equivalent functionality can be achieved via the pretokenizer callback. Following YAGNI, we avoid adding dedicated options with low immediate necessity. This reverts commit bcd4d8c92fa667df2e560fc84291331ce355bf0e.

Commit:bcd4d8c
Author:Taku Kudo
Committer:Taku Kudo

feat: add --exclude_pua_codepoints to exclude Unicode PUA from the vocabulary Adds a new TrainerSpec option `exclude_pua_codepoints` (default: false). When enabled, Unicode Private Use Area code points (U+E000..U+F8FF, U+F0000..U+FFFFD, U+100000..U+10FFFD) are excluded from the base character set and from all learned subwords, regardless of their frequency in the training corpus. PUA characters listed in `required_chars` or `user_defined_symbols` are still included. Excluded code points are decomposed into UTF-8 bytes when `byte_fallback=true`, otherwise mapped to <unk>. - proto: new field `exclude_pua_codepoints = 55`; `auto_character_coverage` is renumbered to 56 to keep the field numbering consistent with the internal TrainerSpec (Next id: 57) - spm_train: new `--exclude_pua_codepoints` flag - spec_parser: parse/print support (Python kwargs work automatically) - trainer_interface: filter in LoadSentences() and IsValidSentencePiece() - tests: C++ (trainer_interface_test, sentencepiece_trainer_test) and Python - docs: doc/options.md, doc/special_symbols.md - regenerate python/src/sentencepiece/sentencepiece_model_pb2.py

Commit:70ed5fc
Author:Taku Kudo

fix, cleanup, docs: fix empty-sentence removal, remove dead code, and update docs - Fix swap-and-pop index increment when removing empty normalized sentences in TrainerInterface::LoadSentences. - Use InlineVector for the Viterbi DP table in lite::Model::EncodeUnigram and merge duplicate BPE checks in lite::Model::Initialize. - Deduplicate UTF8ToUnicodeOffsets into string_util and remove dead declarations, unused parameters, and redundant build dependencies across src/, lite/, and python/. - Update C++, normalization, CMake, and Lite documentation to match current APIs and benchmarks.

Commit:df18339
Author:Taku Kudo

Promote auto_character_coverage to TrainerSpec and remove fixed_sparse_lambda

Commit:c2fbf86
Author:Taku Kudo

feat,fix,deprecate: allow unk_id=-1 with byte_fallback, fix issue #1308, and retire deprecated flags - feat(trainer): allow unk_id = -1 when byte_fallback is true - fix(processor): ignore null-byte piece as UNUSED when byte_fallback is enabled (issue #1308) - deprecate: consolidate deprecated parameters in TrainerSpec and retire CLI flags

Commit:d29cf00
Author:Taku Kudo

deprecate: consolidate deprecated parameters in TrainerSpec and retire CLI flags

Commit:2f36c31
Author:Taku Kudo
Committer:Taku Kudo

feat(trainer): allow unk_id = -1 when byte_fallback is true When byte_fallback is enabled, out-of-vocabulary characters are decomposed into byte tokens, making the explicit <unk> token optional. - Allow unk_id = -1 in TrainerInterface and ModelInterface when byte_fallback is true. - Emit a warning when training with unk_id = -1 regarding backward compatibility with SentencePiece <= 0.2.2. - Guard against negative IDs in SentencePieceProcessor (Decode, IsUnknown, IsControl). - Guard against unk_id == -1 in Python bindings when emit_unk_piece is requested. - Update --unk_id flag description and proto documentation. - Add unit tests covering model training, encoding, and decoding with unk_id = -1.

Commit:470b7d5
Author:Taku Kudo
Committer:Taku Kudo

cleanup: remove self_test_data and consolidate deprecated fields in proto - Remove SelfTestData message and self_test_sample_size/self_test_data from sentencepiece_model.proto. - Consolidate deprecated fields into reserved tags (6, 12, 13, 50, 51, 52 in TrainerSpec; 4 in ModelProto). - Regenerate python/src/sentencepiece/sentencepiece_model_pb2.py. - Remove runtime self-testing loop on model load in sentencepiece_processor.cc. - Remove internal VerifyOutputsEquivalent methods and tests from ModelInterface and UnigramModel. - Remove self_test_sample_size flag and parsing logic from spm_train_main.cc, spec_parser.h, and trainer_interface.cc. - Update third_party/CMakeLists.txt to replace obsolete esaxx with libsais.

Commit:442a7d7
Author:Taku Kudo

deprecate: Clean up deprecated APIs, unused functions, and legacy implementations

Commit:db65384
Author:Taku Kudo

fixed minor security issue

The documentation is generated from this commit.

Commit:53de765
Author:Taku Kudo

allows to load precomputed seed sentencepieces for unigram from a file.

Commit:e58bb68
Author:Taku Kudo

add pretokenization_delimiter options. Initialize seed pieces more accurately.

Commit:9f3ed99
Author:Taku Kudo

Sync internal to github. DP related features are added.

Commit:fab966a
Author:Taku Kudo

sync from internal

Commit:3a5bc58
Author:Taku Kudo

Revert "sync from internal" This reverts commit 05db0894d8ea44b203c3501306061cde9e42c48e.

Commit:05db089
Author:Taku Kudo

sync from internal

Commit:8eaa672
Author:Taku Kudo

change the type of input_sentence_size from int32 to uint64

Commit:f7bc3db
Author:Taku Kudo

merges internal changes to github

Commit:8b921ac
Author:Taku Kudo

Revert the default size of piece length.

Commit:329383b
Author:Taku Kudo

Initial release of 0.19. Merged internal sentencepiece.

Commit:61d59db
Author:Sentencepiece Team
Committer:Keith Stevens

Project import generated by Copybara. PiperOrigin-RevId: 251772401

Commit:59b48eb
Author:Taku Kudo

added --treat_whitespace_as_suffix option to make _ be a suffix of word.

Commit:7b19d68
Author:Taku Kudo

use builtin protobuf-lite package in third_party

Commit:904edfe
Author:Taku Kudo

deperecated mining_sentence_size and training_sentence_size. Load all sentences by default.

Commit:5f635d0
Author:Taku Kudo

support to change the piece of unk/bos/eos/pad

Commit:55e4da4
Author:Taku Kudo

added --max_sentence_length flag

Commit:6d0fd75
Author:Taku Kudo

added --split_by_number flag

Commit:b40cca7
Author:Taku Kudo

Added --use_all_vocab=true flag for WORD/CHAR model

Commit:b6a74ee
Author:Taku Kudo

Added self testing feature.

Commit:5dac483
Author:Taku Kudo

Added --unk_surface option to allow user to change unknown surface string.

Commit:e437e30
Author:Taku Kudo

Support vocab restriction feature

Commit:f228e55
Author:Taku Kudo

Reimplement Trainer with Proto reflection

Commit:ca8754b
Author:Taku Kudo

Add --hard_vocab_limit flag.

Commit:ecbd55a
Author:Taku Kudo
Committer:GitHub

Merge pull request #53 from google/sr Support to change ids of <unk>, <s>, </s>

Commit:d102897
Author:Taku Kudo

Support to change ids of <unk>, <s>, </s>

Commit:ea41f40
Author:Taku Kudo
Committer:GitHub

Merge pull request #48 from tetsuok/fix-default-vocab-size Fix inconsistent default vocab size

Commit:9f3bc53
Author:Tetsuo Kiso

Fix typo in examples of control symbols

Commit:ca20a5a
Author:Tetsuo Kiso

Fix inconsistent default vocab size

Commit:c6a1a19
Author:Taku Kudo

Add Sample/NBestEncode

Commit:2928ce5
Author:Taku Kudo

Initialize repository