Proto commits in yorkie-team/yorkie

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2025-05-09

Remove deprecated MinSyncedTicket and MaxCreatedAtMapByActor (#1208) This commit removes legacy fields and logic related to Lamport timestamp synchronization: - Removed `MinSyncedTicket` and its associated logic - Removed `MaxCreatedAtMapByActor` and related code These were previously retained for backward compatibility during the migration to VersionVectors and are no longer needed.

The documentation is generated from this commit.

2025-05-07

Merge branch 'main' into cleanup-lamport

The documentation is generated from this commit.

2025-04-29

Introduce Document Compaction (#1241) Introduced a document compaction mechanism that resets a document’s history by generating a new initial change from its current final state. This enables safe, incremental migration of documents affected by the removal of backward compatibility for `minSyncedSeq` and `maxCreatedAtMap` (removed in v0.5.2). The compaction process reconstructs the document state, serializes it to YSON, rebuilds it, and replaces historical data with a new ChangePack.

2025-04-25

Add size tracking to CRDT elements Doc size (#1244) Added methods to measure and retrieve the size of document data and metadata, including support for arrays, objects, counters, primitives, text, tree elements, and garbage collection. Additionally, the document summary now provides the document data size, enabling users to have a clearer understanding of the memory usage associated with their documents.

2025-04-18

Enhance document attachment with schema validation

2025-04-17

Merge branch 'main' of https://github.com/yorkie-team/yorkie into cleanup-lamport

Add clients count to project stats (#1236)

Implement CompactDocument feature in ClusterService

2025-04-15

Merge branch 'main' of https://github.com/yorkie-team/yorkie into cleanup-lamport

2025-04-09

Enhance document attachment with schema validation

2025-04-07

Add attached_clients field to ListDocumentSummaries (#1215)

2025-03-31

Remove deprecated maxCreatedAtMapByActor field and related functionality

2025-03-27

Remove deprecated MinSyncedTicket field and related functionality

2025-03-26

Add AllowedOrigins field to Project for CORS support (#1205) This commit introduced a configurable option to specify allowed origins within projects, enabling more robust cross-origin resource sharing management. It also enhanced validation for cross-origin requests to improve security and ensure compliant data access.

2025-03-20

Add project stats retrieval functionality (#1202) - Introduced a new admin API endpoint to provide comprehensive project statistics, including active user metrics and document counts. - Integrated analytics support with a configurable DSN to enhance warehouse functionality and metric conversions.

2025-03-19

Add GetOrCreateUserInfoByGitHubID to manage users by GitHub (#1193) This commit enhanced authentication by integrating GitHub OAuth support. Users can now sign in with GitHub, and profiles display the selected authentication provider.

Add MaxAttachmentsPerDocument to limit Attachments per document (#1196) - Introduced `MaxAttachmentsPerDocument` in project settings. - If the attachment count exceeds the limit, the SDK prevents new attachments and throws `ErrAttachmentsLimitExceeded`. - Utilized `FindClientInfosByAttachedDocRefKey` in the DB client, which leverages the wildcard index `document.$**` in `clients` collection.

2025-03-13

Add MaxSubscribersPerDocument to limit subscriptions per document (#1178) Introduced a limit on the number of concurrent watchDocument subscriptions per document. - Added MaxSubscribersPerDocument to the project. - If the subscription count exceeds the limit, the SDK prevents new subscriptions while allowing push-pull operations to continue.

2025-03-06

Rename connection count limit to attach count limit

Add connection count limit per document to UpdatableProjectFields

2025-02-27

Add connection count limit for cluster server and Update methods for concurrent issue

2025-02-20

Add Event Webhook Support with DocRootChanged Event (#1156) Implements event webhook functionality with configurable URLs and event types at the project level. Introduces `DocRootChanged` event to enable accurate document update tracking, while updating database schema, admin service, and CLI to support webhook configuration.

2025-02-12

Add Kafka Producer for Analytics and User Metrics (#1143) Integrates Kafka-based event publishing system to track Monthly Active Users and other analytics metrics. Implements configurable producer settings with async mode enabled and maintains default values for batch size and retry attempts, while adding new CLI options for Kafka configuration. --------- Co-authored-by: Youngteac Hong <susukang98@gmail.com>

2024-11-28

Add max concurrent connection field to limit max clients per document

2024-10-23

Introduce VersionVector (#1047) This change introduces Lamport Synced Version Vector to resolve defects in the existing garbage collection system that used syncedSeq. Key improvements include: - Added Version Vector implementation with Lamport timestamp support - Implemented database storage and update mechanisms for version vectors - Created min version vector computation for safe garbage collection - Added handling for detached client's version vectors to prevent memory leaks - Updated change ID generation to incorporate version vector information The Version Vector ensures all changes are properly synchronized across replicas before garbage collection occurs, improving system reliability and reducing memory waste from detached clients. --------- Co-authored-by: Youngteac Hong <susukang98@gmail.com>

2024-10-19

Remove `jstype=string` from resources.proto (#1040)

2024-10-18

Detach documents when client is deactivated (#1036) This commit implements the detachment of documents attached to clients upon client deactivation. Previously, the status of the document was only changed to 'detach' within DB.clients.ClientDocInfo.Status, leading to issues where the deactivated client's presence remained active and presence change events were not communicated to other clients. With this change, on client deactivation, the document's packs.PushPull method is called directly to execute the complete detachment process. Moreover, since the Yorkie cluster sharding mechanism assigns specific documents to specific servers, the server managing client deactivation may not always be the one handling the document. Therefore, in such cases, a request will be made to the server currently responsible for the document to detach it properly. To facilitate this communication across the cluster, a new ClusterService has been added. Follow-up tasks that need to be addressed after this PR include: - Implementing authentication for cluster communication and preventing external exposure. - Eliminating redundant database resource requests between clients.Deactivate and server.DetachDocument. - Improving the p.Clear ChangePack creation logic. - Introducing a connection pool for clusterClient. --------- Co-authored-by: raararaara <raararaara@gmail.com>

2024-10-07

Support basic account action for admin (#934) Add the admin APIs ChangePassword and DeleteAccount to enhance the functionality and provide basic account actions for administrators themselves.

Rename SetByIndex to ArraySet (#995) Related to #985

Add Concurrency Tests for Array Operations (#985) A. Total Tests: 49 cases(7*7) Initially, the symmetric relationship between clients in operation combinations was not included. However, since the Lamport clock comparison varies depending on the client ID, the tests were modified to check all cases. B. Operations per Client: 7(2+3+1+1) - Insert: 2 cases(Prev, Prev.Next) - Move: 3 cases(Prev, Prev.Next, Target) - Set: 1 case(Target) - Remove: 1 case(Target) C. Failure Cases: 10 cases(2*2 + 2 + 1*2 + 1*2) 1. *.Prev == Move.Target 4 cases(2*2, symmetric) - Insert.Prev == Move.Target - Move.Prev == Move.Target 2. *.Prev.Next == Set.Target: 2 cases(asymmetric) - Insert.Prev.Next == Set.Target - Move.Prev.Next == Set.Target 3. Move.Target == Set.Target: 2 cases(1*2, symmetric, client ID matters) 4. Remove.Target == Set.Target: 2 cases (1*2, symmetric)

Show Server Version in Yorkie CLI (#938) This commit adds the functionality to display the version of connected Yorkie server in the version command. The version information can be viewed by converting it into yaml or json format using the --output, -o flag. This flag accepts json and yaml strings and converts them to the appropriate format for display. In order to solely check the version of the client CLI without considering the server's version, the --client flag has been included. The development of this feature was inspired by examining the kubectl.

Remove RPC system server

2024-09-30

Introduce system client

2024-09-24

Fix lint proto

Fix lint proto

Fix lint

Add system server to handle perform housekeeping deactivation

2024-09-05

Update Protocol

2024-09-03

Rename SetByIndex to ArraySet (#995) Related to #985

2024-09-02

Add Concurrency Tests for Array Operations (#985) A. Total Tests: 49 cases(7*7) Initially, the symmetric relationship between clients in operation combinations was not included. However, since the Lamport clock comparison varies depending on the client ID, the tests were modified to check all cases. B. Operations per Client: 7(2+3+1+1) - Insert: 2 cases(Prev, Prev.Next) - Move: 3 cases(Prev, Prev.Next, Target) - Set: 1 case(Target) - Remove: 1 case(Target) C. Failure Cases: 10 cases(2*2 + 2 + 1*2 + 1*2) 1. *.Prev == Move.Target 4 cases(2*2, symmetric) - Insert.Prev == Move.Target - Move.Prev == Move.Target 2. *.Prev.Next == Set.Target: 2 cases(asymmetric) - Insert.Prev.Next == Set.Target - Move.Prev.Next == Set.Target 3. Move.Target == Set.Target: 2 cases(1*2, symmetric, client ID matters) 4. Remove.Target == Set.Target: 2 cases (1*2, symmetric)

2024-08-19

Revert "Add systemService to handle perform background deactivation" This reverts commit 0aae9a03e24ac30df254d9aaaa6d66606d1f13b7.

Add systemService to handle perform background deactivation

2024-08-09

Show Server Version in Yorkie CLI (#938) This commit adds the functionality to display the version of connected Yorkie server in the version command. The version information can be viewed by converting it into yaml or json format using the --output, -o flag. This flag accepts json and yaml strings and converts them to the appropriate format for display. In order to solely check the version of the client CLI without considering the server's version, the --client flag has been included. The development of this feature was inspired by examining the kubectl.

2024-07-27

Support basic account action for admin (#934) Add the admin APIs ChangePassword and DeleteAccount to enhance the functionality and provide basic account actions for administrators themselves.

2024-07-19

Enhance GetDocuments API by adding bulk retrieval (#931) This commit enhances document retrieval by adding `FindDocInfosByKeys` to find document information based on given keys for `GetDocuments API`. And it also introduces `include_snapshot` field to specify whether to include snapshots in `GetDocuments API`.

Apply VersionVector to GC

2024-07-16

Introduce VersionVector to detect the relationship between changes

2024-06-28

Add GetDocuments API returning document summaries (#909)

2024-06-04

Add RHTNode removal to converter for consistency (#888) This commit addresses the missing `isRemoved` encoding in the RHT. Similar to other CRDTs like ElementRHT, including tombstone nodes like `isRemoved` during encoding is crucial. However, the RHT did not include tombstone nodes in its encoding, leading to inconsistencies in snapshots.

2024-05-08

Handle concurrent editing and styling in Tree (#854) We will use maxCreatedAtMapByActor until the introduction of VersionVector. https://github.com/yorkie-team/yorkie/pull/800

2024-02-15

Rename vector clocks and Fix gc_test Terminology - change's vector clock -> vector clock - change's latest vector clock -> version vector - each document's vector clock -> version vector - min synced vector clock -> synced version vector - snapshot's vector clock -> latest version vector

2024-02-14

Add Calculate svm in server This approach is silly. - slow and behave like client. - Server track all logs of clients, so we don't need calculate min synced vector every time.

Delete Synced Vector Map

2024-02-12

Fix snapshot just return the latestVectorClock not SVM 1. Now snapshot save latest vector clock not SVM. 2. Clients build their own SVM using the latest VectorClock in snapshots. - So client's SVM is not latest state, it doesn't know peer's vector clock. It need more check - This is due to the overhead of storing and transferring SVMs. #789

Add deleting vectorclock of detaced Actor from synced vector map 1. Now Change has detach flag. It used to delete vectorclock of detached Actor.

2024-02-08

Lint and Structural change

Add synced vector map initialization using snapshots 1. server store synced vector map in snapshot_info. 2. The client also receives vector clock when pulling pack from the snapshot, and replace it's vector map with pulled vector map.

Replace latestCreatedAtMapByActor with vector clock 1. replace text edit's latestCreatedAtMapByActor with vector clock. 2. Fix gc timing in gc_test.go, because gc changed to use vectorclock 3. Because it is used in text edit, context includes current vector.

Add vector clock 1. Add synced vector map in document. 2. Add self vector clock in Change 3. Modify gc to work with min synced vector

2024-01-22

Merge branch 'main' into introduce-sharding-rules-to-clients-collection

2024-01-19

Introduce MongoDB sharding rules to Document-wide collections (#734) This commit introduces MongoDB sharding rules to documents and document-wide collections (e.g. Changes, SyncedSync, Snapshots). Changes: - Introduce RefKey to represent ID and shard key together instead of ID. - Change the reference key of Users from _id to username. - Introduce encoders for types.ID, time.ActorID and ClientDocInfo to MongoDB.

Introduce MongoDB sharding rules to clients collection

Implement Tree.RemoveStyle (#748) This commit added tests for Tree behavior in concurrency scenarios: Styles by 2 users (4 * 6 * 6 = 144 cases): - Ranges(4) - A = B - A contains B - A and B are intersecting - not intersecting - Operations for both A and B(6) - `RemoveStyle({'bold'})` - `Style({'bold': 'aa'})` - `Style({'bold': 'bb'})` - `RemoveStyle({'italic'})` - `Style({'italic': 'aa'})` - `Style({'italic': 'bb'})` Edit and Style (6 * 5 * 2 = 60 cases): - Ranges(6) - A = B - A contains B - B contains A - A and B are intersecting - B is next to A - A is next to B. - Operations for A(5) - Insert front of range - Insert back of range - Insert middle of range - Delete - Change - Operations for B(2) - `RemoveStyle({'bold'})` - `Style({'bold': 'aa'})`

2024-01-17

Add Document type

2023-12-15

Migrate RPC to ConnectRPC (#703) ConnectRPC supports following features: - Supports interoperation and multiplexing of grpc, grpc-web, and Connect protocols. - Supports language-familiar primitives like net/http and fetch. - Supports simple and no-boilerplate codes. With this support, we can resolve several issues: - Removes dependency with Envoy proxy to communicate with JS SDK web client by grpc-web. - Removes dependency with deprecated gogo/protobuf. - Supports JSON based REST API which will satisfy these needs.

2023-12-04

Implement splitLevel of Tree.Edit (#705) When a user positions the cursor within a paragraph and hits the enter key, the paragraph is divided into two separate paragraphs: - Original: `<p>a|b</p>` - Result: `<p>a</p><p>b</p>` In this commit, `splitLevel` is introduced to support this scenario. ```ts doc.update((root) => { root.tree.edit([0,0], {type:'p',children:[{type:'text', value: 'ab'}]}); root.tree.edit([2,2], undefined, 1); // split the paragraph }); ``` This commit does not cover all test cases that may arise due to the split operation. This is because we need to verify the interface first. we need to add test cases in the simultaneous editing situation to verify the algorithm later.

2023-11-28

Change ref key of Clients from _id into (key, _id)

2023-11-22

Support document paging using (_id, key) order

Change ref key of Documents from _id into (key, _id)

2023-10-02

Introduce Broadcast API (#631) This commit introduces a "broadcast" feature for sharing general events. Key implementations include the Document.Broadcast API, SDK enhancements, and server-side additions. Test cases cover broadcasting for unsubscribed events, payload serialization to JSON, and document watching during broadcast errors. --------- Co-authored-by: Youngteac Hong <susukang98@gmail.com>

2023-09-25

Remove createdAtMapByActor in EditReverseOperation

Replace TextNodeIDWithLength to RGATreeSplitPos

2023-09-18

Add fields markOpsBefore and markOpsAfter to TextNode in protobuf

2023-09-15

Merge branch 'main' into feat/text-edit-reverse

Add boundary type 'none'

Update protocol

2023-09-13

Update protocol buffer to add edit_reverse Op

Support concurrent formatting of Text (#639) Currently, we are unable to check for concurrent cases when applying the Text.setStyle operation. This pull request introduces a map called latestCreatedAtMapByActor to track the causality between the operations of the two clients and ensures that the results converge into one.

2023-09-11

Add edit_reverse operation for js-sdk test

2023-08-21

Change TreeNode to have IDs instead of insPrev, insNext (#622) During the conversion, it's hard to find insPrev, insNext nodes from IDs because we can't use LLRB at that moment. Therefore, this commit changed it to keep IDs of insPrev and insNext and find the actual nodes when we need it. Change Tree.Prepend to call updateAncestorSize only when a node is not removed during message conversion of pb to model. --------- Co-authored-by: Hackerwins <susukang98@gmail.com>

2023-08-18

Cleanup proto (#614) * Remove unused client id(key) from API response * Change clientID type from bytes to string in the API

Change 'Documents' from plural to singular in DocEvent (#613) * Change 'Documents' from plural to singular in DocEvent * Add some comments

Handle concurrent editing of Tree.Edit (#607) Introduced new logical timestamp for identifying the position in local/remote editing, and ensures commutative editing in concurrent cases. This logical timestamp consists of {parentID, leftSiblingID}. This allows editing at the front of text nodes by using a reference to the parent, as well as getting rid of the local offset used to access an element node's children previously by using the leftSiblingID. --------- Co-authored-by: MoonGyu1 <s201801402@hufs-gsuite.kr> Co-authored-by: sejongk <sepaper@naver.com>

2023-08-16

Revise protobuf

2023-08-15

Clean up protobuf file

Introduce latestCreatedAtMapByActor for concurrent insertion and deletion

2023-08-14

Restore comment about Select in proto

Remove indent in front of lines

Apply the naming convention in protobuf

2023-08-09

Update API resource

2023-08-03

Clean up code (#595) - Remove unused functions and variables - Remove redundant type casting - Add comments to describe deprecated operation

2023-07-18

Move Presence from Client to Document (#582) To ensure atomic delivery of data for `Document` and `Presence` to users, we are removing `Presence` from the `Client` and moving it to the `Document`. In the future, we plan to further modify it by changing `Document` to `Channel` or `Room` and rename `Root` as `Document`. UpdatePresence `UpdatePresence API` has been removed. Now, document and presence changes are transmitted using the `PushPull API`. Consequently, the presence is no longer transmitted as a component of the `Client`, but rather included within the change. ChangePack now contains `presence`. WatchDocument In the `WatchDocument API`, both the request and response only include the client ID, excluding the presence. --------- Co-authored-by: Youngteac Hong <susukang98@gmail.com>

2023-07-13

Allow users to pass multi nodes when calling Tree.edit (#579) * Allow users to pass multi nodes when calling Tree.edit * Add nodes validation to Tree.edit

2023-07-06

Add include-snapshot flag to ListDocuments API (#575) Co-authored-by: YoonKiJin <ykjin5715@gmail.com>

2023-07-04

Merge branch 'main' of https://github.com/yorkie-team/yorkie into presence-batch

2023-07-03

Add RemoveIfNotAttached flag to Detach (#560) Co-authored-by: Hackerwins <susukang98@gmail.com>

2023-06-28

Rename PresenceInfo to Presence

2023-06-27

Remove clock infomation from presence Since presence is included in Change, there is no change for the order to be reversed when it is received by other peers.

2023-06-26

Merge branch 'main' of https://github.com/yorkie-team/yorkie into presence-batch

2023-06-22

Add force flag to RemoveDocument command (#558) * Add force flag to RemoveDocument command After this commit, an error occurred when attempting to remove a document that was attached to other clients. To forcibly remove a document, users can now use the "--force" flag. Additionally, this commit modifies the locker's condition for "AttachDocument" and "RemoveDocument" to "always" instead of "pack.HasChanges()". * Extract duplicated tests between memDB and MongoDB --------- Co-authored-by: Youngteac Hong <susukang98@gmail.com>