package binlogdata

Get desktop application:
View/edit binary Protocol Buffers messages

BinlogSource specifies the source and filter parameters for Filtered Replication. KeyRange and Tables are legacy. Filter is the new way to specify the filtering rules.

Used in: tabletmanagerdata.CreateVReplicationWorkflowRequest, tabletmanagerdata.ReadVReplicationWorkflowResponse.Stream, vtctldata.Workflow.Stream

• string keyspace = 1

  the source keyspace

• string shard = 2

  the source shard

• topodata.TabletType tablet_type = 3

  the source tablet type

• optional topodata.KeyRange key_range = 4

  KeyRange is set if the request is for a keyrange

• repeated string tables = 5

  Tables is set if the request is for a list of tables

• optional Filter filter = 6

  Filter is set if we're using the generalized representation for the filter.

• OnDDLAction on_ddl = 7

  OnDdl specifies the action to be taken when a DDL is encountered.

• string external_mysql = 8

  Source is an external mysql. This attribute should be set to the username to use in the connection

• bool stop_after_copy = 9

  StopAfterCopy specifies if vreplication should be stopped after copying is done.

• string external_cluster = 10

  ExternalCluster is the name of the mounted cluster which has the source keyspace/db for this workflow it is of the type <cluster_type.cluster_name>

• string source_time_zone = 11

  SourceTimeZone is the time zone in which datetimes on the source were stored, provided as an option in MoveTables

• string target_time_zone = 12

  TargetTimeZone is not currently specifiable by the user, defaults to UTC for the forward workflows and to the SourceTimeZone in reverse workflows

BinlogTransaction describes a transaction inside the binlogs. It is streamed by vttablet for filtered replication, used during resharding.

Used in: StreamKeyRangeResponse, StreamTablesResponse

• repeated BinlogTransaction.Statement statements = 1

  the statements in this transaction

• optional query.EventToken event_token = 4

  The Event Token for this event.

Used in: BinlogTransaction

• Statement.Category category = 1

  what type of statement is this?

• optional Charset charset = 2

  charset of this statement, if different from pre-negotiated default.

• bytes sql = 3

  the sql

Used in: Statement

• BL_UNRECOGNIZED = 0

• BL_BEGIN = 1

• BL_COMMIT = 2

• BL_ROLLBACK = 3

• BL_DML_DEPRECATED = 4

  BL_DML is deprecated.

• BL_DDL = 5

• BL_SET = 6

• BL_INSERT = 7

• BL_UPDATE = 8

• BL_DELETE = 9

Charset is the per-statement charset info from a QUERY_EVENT binlog entry.

Used in: BinlogTransaction.Statement, StreamKeyRangeRequest, StreamTablesRequest

• int32 client = 1

  @@session.character_set_client

• int32 conn = 2

  @@session.collation_connection

• int32 server = 3

  @@session.collation_server

CharsetConversion represent a conversion of text from one charset to another

Used in: Rule

• string from_charset = 1

  FromCharset is the charset name from which we convert the text (e.g. latin1)

• string to_charset = 2

  ToCharset is the charset name to which we convert the text (e.g. utf8mb4)

FieldEvent represents the field info for a table.

Used in: VEvent

• string table_name = 1

• repeated query.Field fields = 2

• string keyspace = 3

• string shard = 4

• bool enum_set_string_values = 25

  Are ENUM and SET field values already mapped to strings in the ROW events? This allows us to transition VTGate VStream consumers from the pre v20 behavior of having to do this mapping themselves to the v20+ behavior of not having to do this anymore and to expect string values directly. NOTE: because this is the use case, this is ONLY ever set today in vstreams managed by the vstreamManager.

• bool is_internal_table = 26

  set for sidecardb tables

Filter represents a list of ordered rules. The first match wins.

Used in: BinlogSource, VStreamRequest, vtgate.VStreamRequest

• repeated Rule rules = 1

• Filter.FieldEventMode field_event_mode = 2

  FieldEventMode specifies the behavior if there is a mismatch between the current schema and the fields in the binlog. This can happen if the binlog position is before a DDL that would cause the fields to change. If vstreamer detects such an inconsistency, the behavior depends on the FieldEventMode. If the value is ERR_ON_MISMATCH (default), then it errors out. If it's BEST_EFFORT, it sends a field event with fake column names as "@1", "@2", etc.

• int64 workflow_type = 3

• string workflow_name = 4

Used in: Filter

• ERR_ON_MISMATCH = 0

• BEST_EFFORT = 1

Journal contains the metadata for a journal event. The commit of a journal event indicates the point of no return for a migration.

Used in: VEvent

• int64 id = 1

  Id represents a unique journal id.

• MigrationType migration_type = 2

• repeated string tables = 3

  Tables is set if the journal represents a TABLES migration.

• string local_position = 4

  LocalPosition is the source position at which the migration happened.

• repeated ShardGtid shard_gtids = 5

  ShardGtids is the list of targets to which the migration took place.

• repeated KeyspaceShard participants = 6

  Participants is the list of source participants for a migration. Every participant is expected to have an identical journal entry. While streaming, the client must wait for the journal entry to be received from all pariticipants, and then replace them with new streams specified by ShardGtid. If a stream does not have all participants, a consistent migration is not possible.

• repeated string source_workflows = 7

  SourceWorkflows is the list of workflows in the source shard that were migrated to the target. If a migration fails after a Journal is committed, this information is used to start the target streams that were created prior to the creation of the journal.

KeyspaceShard represents a keyspace and shard.

Used in: Journal

• string keyspace = 1

• string shard = 2

Used in: VEvent

• optional TableLastPK table_last_p_k = 1

• bool completed = 2

MigrationType specifies the type of migration for the Journal.

Used in: Journal

• TABLES = 0

• SHARDS = 1

• repeated MinimalTable tables = 1

Used in: MinimalSchema

• string name = 1

• repeated query.Field fields = 2

• repeated int64 p_k_columns = 3

• string p_k_index_name = 4

  This will be PRIMARY when the actual primary key is used and it will be the name of the Primary Key equivalent if one is used instead. Otherwise it will be empty.

OnDDLAction lists the possible actions for DDLs.

Used in: BinlogSource, tabletmanagerdata.UpdateVReplicationWorkflowRequest

• IGNORE = 0

• STOP = 1

• EXEC = 2

• EXEC_IGNORE = 3

RowChange represents one row change. If Before is set and not After, it's a delete. If After is set and not Before, it's an insert. If both are set, it's an update.

Used in: RowEvent

• optional query.Row before = 1

• optional query.Row after = 2

• optional RowChange.Bitmap data_columns = 3

  DataColumns is a bitmap of all columns: bit is set if column is present in the after image.

• optional RowChange.Bitmap json_partial_values = 4

  JsonPartialValues is a bitmap of any JSON columns, where the bit is set if the value in the AFTER image is a partial JSON value that is represented as an expression of JSON_[INSERT|REPLACE|REMOVE](%s, '$.path', value) which then is used to add/update/remove a path in the JSON document. When the value is used the fmt directive must be replaced by the actual column name of the JSON field.

Used in: RowChange

• int64 count = 1

• bytes cols = 2

RowEvent represent row events for one table.

Used in: VEvent

• string table_name = 1

• repeated RowChange row_changes = 2

• string keyspace = 3

• string shard = 4

• uint32 flags = 5

  https://dev.mysql.com/doc/dev/mysql-server/latest/classbinary__log_1_1Rows__event.html

• bool is_internal_table = 6

  set for sidecardb tables

Rule represents one rule in a Filter.

Used in: Filter, tabletmanagerdata.UpdateVReplicationWorkflowRequest

• string match = 1

  Match can be a table name or a regular expression. If it starts with a '/', it's a regular expression. For example, "t" matches a table named "t", whereas "/t.*" matches all tables that begin with 't'.

• string filter = 2

  Filter: If empty, all columns and rows of the matching tables are sent. If it's a keyrange like "-80", only rows that match the keyrange are sent. If Match is a table name instead of a regular expression, the Filter can also be a select expression like this: "select * from t", same as an empty Filter, or "select * from t where in_keyrange('-80')", same as "-80", or "select col1, col2 from t where in_keyrange(col1, 'hash', '-80'), or What is allowed in a select expression depends on whether it's a vstreamer or vreplication request. For more details, please refer to the specific package documentation. On the vreplication side, Filter can also accept a special "exclude" value, which will cause the matched tables to be excluded. TODO(sougou): support this on vstreamer side also.

  ConvertEnumToText: optional, list per enum column name, the list of textual values. When reading the binary log, all enum values are numeric. But sometimes it is useful/needed to know what the textual mapping are. Online DDL provides such use case.

• map<string, string> convert_enum_to_text = 3

  Example: key="color", value="'red','green','blue'"

• map<string, CharsetConversion> convert_charset = 4

  ConvertCharset: optional mapping, between column name and a CharsetConversion. This hints to vreplication that columns are encoded from/to non-trivial charsets The map is only populated when either "from" or "to" charset of a column are non-trivial trivial charsets are utf8 and ascii variants.

• string source_unique_key_columns = 5

  SourceUniqueKeyColumns represents the ordered columns in the index used by rowstreamer to iterate the table It is comma delimited, as in col1,col2,col3 (tokens are escaped via net/url)

• string target_unique_key_columns = 6

  TargetUniqueKeyColumns represents the ordered columns in that index used by vcopier and vplayer to apply rows It is comma delimited, as in col1,col2,col3 (tokens are escaped via net/url)

• string source_unique_key_target_columns = 7

  SourceUniqueKeyTargetColumns represents the names of columns in target table, mapped from the chosen unique key on source tables (some columns may be renamed from source to target)

• map<string, bool> convert_int_to_enum = 8

  ConvertIntToEnum lists any columns that are converted from an integral value into an enum. such columns need to have special transofrmation of the data, from an integral format into a string format. e.g. the value 0 needs to be converted to '0'.

• string force_unique_key = 9

  ForceUniqueKey gives vtreamer a hint for `FORCE INDEX (...)` usage.

ShardGtid contains the GTID position for one shard. It's used in a request for requesting a starting position. It's used in a response to transmit the current position of a shard. It's also used in a Journal to indicate the list of targets and shard positions to migrate to.

Used in: Journal, VGtid

• string keyspace = 1

• string shard = 2

• string gtid = 3

• repeated TableLastPK table_p_ks = 4

StreamKeyRangeRequest is the payload to StreamKeyRange

Used as request type in: binlogservice.UpdateStream.StreamKeyRange

• string position = 1

  where to start

• optional topodata.KeyRange key_range = 2

  what to get

• optional Charset charset = 3

  default charset on the player side

StreamKeyRangeResponse is the response from StreamKeyRange

Used as response type in: binlogservice.UpdateStream.StreamKeyRange

• optional BinlogTransaction binlog_transaction = 1

StreamTablesRequest is the payload to StreamTables

Used as request type in: binlogservice.UpdateStream.StreamTables

• string position = 1

  where to start

• repeated string tables = 2

  what to get

• optional Charset charset = 3

  default charset on the player side

StreamTablesResponse is the response from StreamTables

Used as response type in: binlogservice.UpdateStream.StreamTables

• optional BinlogTransaction binlog_transaction = 1

Used in: LastPKEvent, ShardGtid, VStreamRequest

• string table_name = 1

• optional query.QueryResult lastpk = 3

VEvent represents a vstream event. A FieldEvent is sent once for every table, just before the first event for that table. The client is expected to cache this information and match it against the RowEvent which contains the table name. A GTID event always precedes a commitable event, which can be COMMIT, DDL or OTHER. OTHER events are non-material events that have no additional metadata.

Used in: VStreamResponse, vtgate.VStreamResponse

• VEventType type = 1

• int64 timestamp = 2

  Timestamp is the binlog timestamp in seconds. The value should be ignored if 0.

• string gtid = 3

  Gtid is set if the event type is GTID.

• string statement = 4

  Statement is set if the event type is DDL, DML or SAVEPOINT.

• optional RowEvent row_event = 5

  RowEvent is set if the event type is ROW.

• optional FieldEvent field_event = 6

  FieldEvent is set if the event type is FIELD.

• optional VGtid vgtid = 7

  Vgtid is set if the event type is VGTID. This event is only generated by VTGate's VStream function.

• optional Journal journal = 8

  Journal is set if the event type is JOURNAL.

• string dml = 9

  Dml is set if the event type is INSERT, REPLACE, UPDATE or DELETE.

• int64 current_time = 20

  CurrentTime specifies the current time when the message was sent. This can be used to compenssate for clock skew.

• optional LastPKEvent last_p_k_event = 21

  LastPK is the last PK for a table

• string keyspace = 22

  the source keyspace

• string shard = 23

  the source shard

• bool throttled = 24

  indicate that we are being throttled right now

• string throttled_reason = 25

  ThrottledReason is a human readable string that explains why the stream is throttled

VEventType enumerates the event types. Many of these types will not be encountered in RBR mode.

Used in: VEvent

• UNKNOWN = 0

• GTID = 1

• BEGIN = 2

• COMMIT = 3

• ROLLBACK = 4

• DDL = 5

• INSERT = 6

  INSERT, REPLACE, UPDATE, DELETE and SET will not be seen in RBR mode.

• REPLACE = 7

• UPDATE = 8

• DELETE = 9

• SET = 10

• OTHER = 11

  OTHER is a dummy event. If encountered, the current GTID must be recorded by the client to be able to resume.

• ROW = 12

• FIELD = 13

• HEARTBEAT = 14

  HEARTBEAT is sent if there is inactivity. If a client does not receive events beyond the hearbeat interval, it can assume that it's lost connection to the vstreamer.

• VGTID = 15

  VGTID is generated by VTGate's VStream that combines multiple GTIDs.

• JOURNAL = 16

• VERSION = 17

• LASTPK = 18

• SAVEPOINT = 19

• COPY_COMPLETED = 20

  COPY_COMPLETED is sent when VTGate's VStream copy operation is done. If a client experiences some disruptions before receiving the event, the client should restart the copy operation.

A VGtid is a list of ShardGtids.

Used in: VEvent, vtgate.VStreamRequest

• repeated ShardGtid shard_gtids = 1

VReplicationWorklfowState defines the valid states that a workflow can be in.

Used in: tabletmanagerdata.ReadVReplicationWorkflowResponse.Stream, tabletmanagerdata.ReadVReplicationWorkflowsRequest, tabletmanagerdata.UpdateVReplicationWorkflowRequest, tabletmanagerdata.UpdateVReplicationWorkflowsRequest

• Unknown = 0

• Init = 1

• Stopped = 2

• Copying = 3

• Running = 4

• Error = 5

• Lagging = 6

VReplicationWorkflowSubType define types of vreplication workflows.

Used in: tabletmanagerdata.CreateVReplicationWorkflowRequest, tabletmanagerdata.ReadVReplicationWorkflowResponse

• None = 0

• Partial = 1

• AtomicCopy = 2

VReplicationWorkflowType define types of vreplication workflows.

Used in: tabletmanagerdata.CreateVReplicationWorkflowRequest, tabletmanagerdata.ReadVReplicationWorkflowResponse

• Materialize = 0

• MoveTables = 1

• CreateLookupIndex = 2

• Migrate = 3

• Reshard = 4

• OnlineDDL = 5

Used in: VStreamRequest, VStreamRowsRequest, VStreamTablesRequest

• repeated string internal_tables = 1

• map<string, string> config_overrides = 2

VStreamRequest is the payload for VStreamer

Used as request type in: queryservice.Query.VStream

• optional vtrpc.CallerID effective_caller_id = 1

• optional query.VTGateCallerID immediate_caller_id = 2

• optional query.Target target = 3

• string position = 4

• optional Filter filter = 5

• repeated TableLastPK table_last_p_ks = 6

• optional VStreamOptions options = 7

VStreamResponse is the response from VStreamer

Used as response type in: queryservice.Query.VStream

• repeated VEvent events = 1

VStreamResultsRequest is the payload for VStreamResults The ids match VStreamRows, in case we decide to merge the two. The ids match VStreamRows, in case we decide to merge the two.

Used as request type in: queryservice.Query.VStreamResults

• optional vtrpc.CallerID effective_caller_id = 1

• optional query.VTGateCallerID immediate_caller_id = 2

• optional query.Target target = 3

• string query = 4

VStreamResultsResponse is the response from VStreamResults The ids match VStreamRows, in case we decide to merge the two.

Used as response type in: queryservice.Query.VStreamResults

• repeated query.Field fields = 1

• string gtid = 3

• repeated query.Row rows = 4

VStreamRowsRequest is the payload for VStreamRows

Used as request type in: queryservice.Query.VStreamRows

• optional vtrpc.CallerID effective_caller_id = 1

• optional query.VTGateCallerID immediate_caller_id = 2

• optional query.Target target = 3

• string query = 4

• optional query.QueryResult lastpk = 5

• optional VStreamOptions options = 6

VStreamRowsResponse is the response from VStreamRows

Used as response type in: queryservice.Query.VStreamRows

• repeated query.Field fields = 1

• repeated query.Field pkfields = 2

• string gtid = 3

• repeated query.Row rows = 4

• optional query.Row lastpk = 5

• bool throttled = 6

  Throttled indicates that rowstreamer is being throttled right now

• bool heartbeat = 7

  Heartbeat indicates that this is a heartbeat message

• string throttled_reason = 8

  ThrottledReason is a human readable string that explains why the stream is throttled

VStreamTablesRequest is the payload for VStreamTables

Used as request type in: queryservice.Query.VStreamTables

• optional vtrpc.CallerID effective_caller_id = 1

• optional query.VTGateCallerID immediate_caller_id = 2

• optional query.Target target = 3

• optional VStreamOptions options = 4

VStreamTablesResponse is the response from VStreamTables

Used as response type in: queryservice.Query.VStreamTables

• string table_name = 1

• repeated query.Field fields = 2

• repeated query.Field pkfields = 3

• string gtid = 4

• repeated query.Row rows = 5

• optional query.Row lastpk = 6