package ml_metadata
Get desktop application:
View/edit binary Protocol Buffers messages
LINT.IfChange
Inserts or updates an ArtifactType. A type has a set of strong typed properties describing the schema of any stored instance associated with that type. A type is identified by a name and an optional version. Type Creation: If no type exists in the database with the given identifier (name, version), it creates a new type and returns the type_id. Type Evolution: If the request type with the same (name, version) already exists (let's call it stored_type), the method enforces the stored_type can be updated only when the request type is backward compatible for the already stored instances. Backwards compatibility is violated iff: a) there is a property where the request type and stored_type have different value type (e.g., int vs. string) b) `can_add_fields = false` and the request type has a new property that is not stored. c) `can_omit_fields = false` and stored_type has an existing property that is not provided in the request type. If non-backward type change is required in the application, e.g., deprecate properties, re-purpose property name, change value types, a new type can be created with a different (name, version) identifier. Note the type version is optional, and a version value with empty string is treated as unset. Args: artifact_type: the type to be inserted or updated. can_add_fields: when set to true, new properties can be added; when set to false, returns ALREADY_EXISTS if the request type has properties that are not in stored_type. can_omit_fields: when set to true, stored properties can be omitted in the request type; when set to false, returns ALREADY_EXISTS if the stored_type has properties not in the request type. Returns: The type_id of the stored type. Raises: ALREADY_EXISTS error in the case listed above. INVALID_ARGUMENT error, if the given type has no name, or any property value type is unknown.
message
metadata_store_service.proto:168optional artifact_type = 1
The field is required in any request. Stored types in MLMD can be updated by introducing new properties and remain backward compatible. If a type with the same name exists in the database, it updates the existing type, otherwise it creates a new type.
optional bool can_add_fields = 2
If true then allows adding properties to an existing stored type. If false, then type update is not allowed and it raises AlreadyExists error if the given type has any new property that is not defined in the stored type.
optional bool can_omit_fields = 5
If true then allows omitting properties of an existing stored type. If false, then no properties of the stored type can be omitted in the given type, otherwise it raises AlreadyExists error.
optional bool can_delete_fields = 3
Deprecated fields.
optional bool all_fields_match = 4
optional transaction_options = 6
Options regarding transactions.
message
metadata_store_service.proto:194optional int64 type_id = 1
The type ID of the artifact type.
Inserts or updates an ExecutionType. Please refer to PutArtifactType for type upsert API description.
message
metadata_store_service.proto:248optional execution_type = 1
The field is required in any request. Stored types in MLMD can be updated by introducing new properties and remain backward compatible. If a type with the same name exists in the database, it updates the existing type, otherwise it creates a new type.
optional bool can_add_fields = 2
If true then allows adding properties to an existing stored type. If false, then type update is not allowed and it raises AlreadyExists error if the given type has any new property that is not defined in the stored type.
optional bool can_omit_fields = 5
If true then allows omitting properties of an existing stored type. If false, then no properties of the stored type can be omitted in the given type, otherwise it raises AlreadyExists error.
optional bool can_delete_fields = 3
Deprecated fields.
optional bool all_fields_match = 4
optional transaction_options = 6
Options regarding transactions.
message
metadata_store_service.proto:274optional int64 type_id = 1
The type ID of the execution type.
Inserts or updates an ContextType. Please refer to PutArtifactType for type upsert API description.
message
metadata_store_service.proto:439optional context_type = 1
The field is required in any request. Stored types in MLMD can be updated by introducing new properties and remain backward compatible. If a type with the same name exists in the database, it updates the existing type, otherwise it creates a new type.
optional bool can_add_fields = 2
If true then allows adding properties to an existing stored type. If false, then type update is not allowed and it raises AlreadyExists error if the given type has any new property that is not defined in the stored type.
optional bool can_omit_fields = 5
If true then allows omitting properties of an existing stored type. If false, then no properties of the stored type can be omitted in the given type, otherwise it raises AlreadyExists error.
optional bool can_delete_fields = 3
Deprecated fields.
optional bool all_fields_match = 4
optional transaction_options = 6
Options regarding transactions.
message
metadata_store_service.proto:465optional int64 type_id = 1
The type ID of the context type.
Bulk inserts types atomically.
message
metadata_store_service.proto:406repeated artifact_types = 1
repeated execution_types = 2
repeated context_types = 3
optional bool can_add_fields = 4
If true then allows adding properties to an existing stored type. If false, then type update is not allowed and it raises AlreadyExists error if the given type has any new property that is not defined in the stored type.
optional bool can_omit_fields = 7
If true then allows omitting properties of an existing stored type. If false, then no properties of the stored type can be omitted in the given type, otherwise it raises AlreadyExists error.
optional bool can_delete_fields = 5
Deprecated fields.
optional bool all_fields_match = 6
optional transaction_options = 8
Options regarding transactions.
message
metadata_store_service.proto:430repeated int64 artifact_type_ids = 1
The type ids of the artifact type.
repeated int64 execution_type_ids = 2
The type ids of the execution type.
repeated int64 context_type_ids = 3
The type ids of the context type.
Inserts or updates artifacts in the database. If an artifact_id is specified for an artifact, it is an update. If an artifact_id is unspecified, it will insert a new artifact. For new artifacts, type must be specified. For old artifacts, type must be unchanged or unspecified. It is not guaranteed that the created or updated artifacts will share the same `create_time_since_epoch` or `last_update_time_since_epoch` timestamps. Args: artifacts: A list of artifacts to insert or update. Returns: A list of artifact ids index-aligned with the input.
message
metadata_store_service.proto:53repeated artifacts = 1
optional options = 2
Additional options to change the behavior of the method.
optional transaction_options = 3
Options regarding transactions.
optional update_mask = 4
FieldMask for artifacts in the PUT update If `artifact.id` is not specified, it means a new artifact will be created and `update_mask` will not be applied to the creation. If `update_mask` is empty, update the artifacts as a whole. If `update_mask` is not empty, only update fields or properties specified in `update_mask`. Example request protos: 1. Examples that update `properties` / `custom_properties`: 1.1 Add a <'key', 'val'> pair into `custom_properties`: { artifacts { id: 1234 type_id: 5678 custom_properties { key: "key" value: { string_value: "val" } } } update_mask { paths: "custom_properties.key" } } 1.2 Set `custom_properties['key'].bool_value` to true: { artifacts { id: 1234 type_id: 5678 custom_properties { key: "key" value: { bool_value: true } } } update_mask { paths: "custom_properties.key" } } 1.3 Delete the complete <'key', 'val'> pair from `custom_properties`: { artifacts { id: 1234 type_id: 5678 custom_properties {} } update_mask { paths: "custom_properties.key" } } 2. Examples that update fields such as `uri`, `external_id`, etc: 2.1 Update `external_id` field: { artifacts { id: 1234 type_id: 5678 external_id: "new_value" } update_mask { paths: "external_id" } } 2.2 Set `uri` field: { artifacts { id: 1234 type_id: 5678 uri: "set_value" } update_mask { paths: "uri" } } If `paths: "properties"` or `paths: "custom_properties"` are added to `update_mask`, the key-level updates will be ignored and we only perform field-level updates on the all `properties`/`custom_properties`. For example: If the mask is: {"properties", "properties.key1"}, the field path "properties.key1" will be ignored and all `properties` will be updated. (Do not suggest) If the mask is {"properties", "external_id"}, all `properties` and field `external_id` will be updated. (Do not suggest)
message
metadata_store_service.proto:163repeated int64 artifact_ids = 1
A list of artifact ids index-aligned with PutArtifactsRequest.
Inserts or updates executions in the database. If an execution_id is specified for an execution, it is an update. If an execution_id is unspecified, it will insert a new execution. For new executions, type must be specified. For old executions, type must be unchanged or unspecified. It is not guaranteed that the created or updated executions will share the same `create_time_since_epoch` or `last_update_time_since_epoch` timestamps. Args: executions: A list of executions to insert or update. Returns: A list of execution ids index-aligned with the input.
message
metadata_store_service.proto:199repeated executions = 1
optional transaction_options = 2
Options regarding transactions.
optional update_mask = 3
FieldMask for executions in the PUT update If `execution.id` is not specified, it means a new execution will be created and `update_mask` will not be applied to the creation. If `update_mask` is empty, update the executions as a whole. If `update_mask` is not empty, only update fields or properties specified in `update_mask`. Example request protos: 1. Add a <'key', 'val'> pair into `custom_properties`: { executions { id: 1234 type_id: 5678 custom_properties { key: "key" value: { string_value: "val" } } } update_mask { paths: "custom_properties.key" } } 2. Set `last_known_state` field: { executions { id: 1234 type_id: 5678 last_known_state: CACHED } update_mask { paths: "last_known_state" } } Please refer to `PutArtifactsRequest` for more details.
message
metadata_store_service.proto:243repeated int64 execution_ids = 1
A list of execution ids index-aligned with PutExecutionsRequest.
Inserts events in the database. The execution_id and artifact_id must already exist. Once created, events cannot be modified. AlreadyExists error will be raised if duplicated events are found. It is not guaranteed that the created or updated events will share the same `milliseconds_since_epoch` timestamps. Args: events: A list of events to insert or update.
message
metadata_store_service.proto:279repeated events = 1
optional transaction_options = 2
Options regarding transactions.
message
metadata_store_service.proto:286(message has no fields)
Inserts or updates an Execution and its input and output artifacts and related contexts atomically. The `artifact_event_pairs` include the state changes of the Artifacts used or generated by the Execution, as well as the input/output Event. The `contexts` describe the associations of the execution and the attributions of the artifacts. If an execution_id is specified, it is an update on the corresponding execution, otherwise it does an insertion. For insertion, type must be specified. Same rule applies to artifacts and contexts in the request. Corresponding errors may raised. For example: AlreadyExists error will be raised if duplicated executions, artifacts or events are found. It is not guaranteed that the created or updated executions, artifacts, contexts and events will share the same `create_time_since_epoch`, `last_update_time_since_epoch`, or `milliseconds_since_epoch` timestamps. Args: execution: An execution to insert or update. artifact_event_pairs: Artifacts to insert or update and events to insert. contexts: The contexts that the execution and the artifacts belong to. Returns: An execution id and a list of artifacts and contexts ids index-aligned with the input.
message
metadata_store_service.proto:288optional execution = 1
The execution that produces many artifact and event pairs.
repeated artifact_event_pairs = 2
The list of artifact and event pairs.
repeated contexts = 3
A list of contexts associated with the execution and artifacts. For each given context without a context.id, it inserts the context, otherwise it updates the stored context with the same id (unless the option force_reuse_context is set). Associations between each pair of contexts and the execution, and attributions between each pair of contexts and artifacts are created if they do not already exist.
optional options = 4
Additional options to change the behavior of the method.
optional transaction_options = 5
Options regarding transactions.
message
metadata_store_service.proto:350optional int64 execution_id = 1
An execution id of the `execution` in PutExecutionRequest.
repeated int64 artifact_ids = 2
A list of artifact ids index-aligned with `artifact_event_pairs` in the PutExecutionRequest.
repeated int64 context_ids = 3
A list of context ids index-aligned with `contexts` in the PutExecutionRequest.
Inserts or updates a lineage subgraph (i.e. a collection of event edges and its executions, artifacts, and related contexts) atomically. The `event_edges` include an Event and the indices of the corresponding execution and artifact from the input list of executions and artifacts. The `contexts` describe the associations of the Execution and the attributions of the Artifact. If an execution_id is specified, it is an update on the corresponding Execution, otherwise it does an insertion. For insertion, type must be specified. These rules apply to Artifacts and Contexts as well. Corresponding errors may be raised. For example: AlreadyExists error will be raised if duplicated executions, artifacts, or events are found. It is not guaranteed that the created or updated executions, artifacts, contexts and events will share the same `create_time_since_epoch`, `last_update_time_since_epoch`, or `milliseconds_since_epoch` timestamps. Args: executions: A list of executions to insert or update. artifacts: A list of artifacts to insert or update. contexts: A list of contexts to insert and/or create associations and attributions with. event_edges: A list of events to insert with the indices of the corresponding execution and artifact from the input lists of executions and artifacts. Returns: Lists of execution, artifact, and context ids index-aligned with the inputs.
message
metadata_store_service.proto:361repeated executions = 1
repeated artifacts = 2
repeated contexts = 3
repeated event_edges = 4
optional options = 5
optional transaction_options = 6
message
metadata_store_service.proto:397repeated int64 execution_ids = 1
A list of execution ids index-aligned with `executions` in the request
repeated int64 artifact_ids = 2
A list of artifact ids index-aligned with `artifacts` in the request
repeated int64 context_ids = 3
A list of context ids index-aligned with `contexts` in the request
Inserts or updates contexts in database and returns a list of context ids. If an context_id is specified for a context, it is an update. If an context_id is unspecified, it will insert a new context. For new contexts, type must be specified. For old contexts, type must be unchanged or unspecified. It is not guaranteed that the created or updated contexts will share the same `create_time_since_epoch` or `last_update_time_since_epoch` timestamps. Args: contexts: A list of contexts to insert or update. Returns: A list of context ids index-aligned with the input.
message
metadata_store_service.proto:470repeated contexts = 1
optional transaction_options = 2
Options regarding transactions.
optional update_mask = 3
FieldMask for contexts in the PUT update If `context.id` is not specified, it means a new context will be created and `update_mask` will not be applied to the creation. If `update_mask` is empty, update the contexts as a whole. If `update_mask` is not empty, only update fields or properties specified in `update_mask`. Example request protos: 1. Add a <'key', 'val'> pair into `custom_properties`: { contexts { id: 1234 type_id: 5678 custom_properties { key: "key" value: { string_value: "val" } } } update_mask { paths: "custom_properties.key" } } 2. Set `name` field: { contexts { id: 1234 type_id: 5678 name: "set_name" } update_mask { paths: "name" } } Please refer to `PutArtifactsRequest` for more details.
message
metadata_store_service.proto:514repeated int64 context_ids = 1
A list of context ids index-aligned with PutContextsRequest.
Inserts attribution and association relationships in the database. The context_id, artifact_id, and execution_id must already exist. If the relationship exists, this call does nothing. Once added, the relationships cannot be modified. Args: attributions: A list of attributions to insert. associations: A list of associations to insert.
message
metadata_store_service.proto:519repeated attributions = 1
repeated associations = 2
optional transaction_options = 3
Options regarding transactions.
message
metadata_store_service.proto:527(message has no fields)
Inserts parental context relationships in the database. The ParentContext relationship has direction. The call fails if cycles are detected. Args: parent_contexts: A list of parent contexts to insert.
message
metadata_store_service.proto:529repeated parent_contexts = 1
optional transaction_options = 2
Options regarding transactions.
message
metadata_store_service.proto:536(message has no fields)
Gets an artifact type. Returns a NOT_FOUND error if the type does not exist.
message
metadata_store_service.proto:659optional string type_name = 1
optional string type_version = 2
If not set, it looks for the type with type_name with default type_version.
optional transaction_options = 3
Options regarding transactions.
message
metadata_store_service.proto:667optional artifact_type = 1
Gets an artifact type, or clear if it does not exist.
Gets a list of artifact types by ID. If no artifact types with an ID exists, the artifact type is skipped.
message
metadata_store_service.proto:847repeated int64 type_ids = 1
optional transaction_options = 2
Options regarding transactions.
message
metadata_store_service.proto:853repeated artifact_types = 1
The result is not index-aligned: if an id is not found, it is not returned.
Gets a list of all artifact types.
message
metadata_store_service.proto:672optional transaction_options = 1
Options regarding transactions.
message
metadata_store_service.proto:677repeated artifact_types = 1
Gets an execution type, or None if it does not exist.
message
metadata_store_service.proto:812optional string type_name = 1
optional string type_version = 2
If not set, it looks for the type with type_name with default type_version.
optional transaction_options = 3
Options regarding transactions.
message
metadata_store_service.proto:820optional execution_type = 1
Gets an execution type, or clear if it does not exist.
Gets a list of execution types by ID. If no execution types with an ID exists, the execution type is skipped.
message
metadata_store_service.proto:859repeated int64 type_ids = 1
optional transaction_options = 2
Options regarding transactions.
message
metadata_store_service.proto:865repeated execution_types = 1
The result is not index-aligned: if an id is not found, it is not returned.
Gets a list of all execution types.
message
metadata_store_service.proto:681optional transaction_options = 1
Options regarding transactions.
message
metadata_store_service.proto:686repeated execution_types = 1
Gets a context type. Returns a NOT_FOUND error if the type does not exist.
message
metadata_store_service.proto:871optional string type_name = 1
optional string type_version = 2
If not set, it looks for the type with type_name with default type_version.
optional transaction_options = 3
Options regarding transactions.
message
metadata_store_service.proto:879optional context_type = 1
Gets a context type, or clear if it does not exist.
Gets a list of context types by ID. If no context types with an ID exists, the context type is skipped.
message
metadata_store_service.proto:884repeated int64 type_ids = 1
optional transaction_options = 2
Options regarding transactions.
message
metadata_store_service.proto:890repeated context_types = 1
The result is not index-aligned: if an id is not found, it is not returned.
Gets a list of all context types.
message
metadata_store_service.proto:690optional transaction_options = 1
Options regarding transactions.
message
metadata_store_service.proto:695repeated context_types = 1
Gets all the artifacts.
message
metadata_store_service.proto:608Request to retrieve Artifacts using List options. If option is not specified then all Artifacts are returned.
optional options = 1
Specify options. Please refer to the documentation of ListOperationOptions for the supported functionalities.
optional transaction_options = 2
Options regarding transactions.
message
metadata_store_service.proto:617repeated artifacts = 1
Returned artifacts.
optional string next_page_token = 2
Token to use to retrieve next page of results if list options are used in the request.
Gets all the executions.
message
metadata_store_service.proto:641Request to retrieve Executions using List options. If option is not specified then all Executions are returned.
optional options = 1
Specify options. Please refer to the documentation of ListOperationOptions for the supported functionalities.
optional transaction_options = 2
Options regarding transactions.
message
metadata_store_service.proto:650repeated executions = 1
Returned executions.
optional string next_page_token = 2
Token to use to retrieve next page of results if list options are used in the request.
Gets all the contexts.
message
metadata_store_service.proto:898Request to retrieve Contexts using List options. If option is not specified then all Contexts are returned.
optional options = 1
Specify options. Please refer to the documentation of ListOperationOptions for the supported functionalities.
optional transaction_options = 2
Options regarding transactions.
message
metadata_store_service.proto:907repeated contexts = 1
Returned contexts.
optional string next_page_token = 2
Token to use to retrieve next page of results if list options are used in the request.
Gets all artifacts with matching ids. The result is not index-aligned: if an id is not found, it is not returned. Args: artifact_ids: A list of artifact ids to retrieve. Returns: Artifacts with matching ids.
message
metadata_store_service.proto:578repeated int64 artifact_ids = 1
A list of artifact ids to retrieve.
optional bool populate_artifact_types = 3
An option to populate all the ArtifactTypes in the response. If true, returns retrieved Artifacts and their artifact types, which can be matched by type_ids. If false, returns only the retrieved Artifacts. Example request proto: { artifact_ids: 101, populate_artifact_types: true, } The response will contain an artifact with id = 101 and an artifact type with id = artifact.type_id().
optional transaction_options = 2
Options regarding transactions.
message
metadata_store_service.proto:597repeated artifacts = 1
Artifacts with matching ids. This is not index-aligned: if an id is not found, it is not returned.
repeated artifact_types = 2
ArtifactTypes populated with matching type_ids owned by `artifacts`. This is not index-aligned: if a type_id is not found, it is not returned.
Gets all executions with matching ids. The result is not index-aligned: if an id is not found, it is not returned. Args: execution_ids: A list of execution ids to retrieve.
message
metadata_store_service.proto:799repeated int64 execution_ids = 1
A list of execution ids to retrieve.
optional transaction_options = 2
Options regarding transactions.
message
metadata_store_service.proto:806repeated executions = 1
The result is not index-aligned: if an id is not found, it is not returned.
Gets all contexts with matching ids. The result is not index-aligned: if an id is not found, it is not returned. Args: context_ids: A list of context ids to retrieve.
message
metadata_store_service.proto:957repeated int64 context_ids = 1
A list of context ids to retrieve.
optional transaction_options = 2
Options regarding transactions.
message
metadata_store_service.proto:964repeated contexts = 1
The result is not index-aligned: if an id is not found, it is not returned.
Gets all the artifacts of a given type.
message
metadata_store_service.proto:538optional string type_name = 1
optional string type_version = 2
If not set, it looks for the type with type_name with default type_version.
optional options = 3
Specify List options. Currently supports: 1. Field to order the results. 2. Page size. If set, the request will first fetch all artifacts with specified `type_name` and `type_version`, then order by a specifield field finally find the correct page and return #Artifacts of the page size. Higher-level APIs may only use the functionalies partially. Please reference the API documentation for the API behaviors.
optional transaction_options = 4
Options regarding transactions.
message
metadata_store_service.proto:557repeated artifacts = 1
optional string next_page_token = 2
Token to use to retrieve next page of results if list options are used in the request.
Gets all the executions of a given type.
message
metadata_store_service.proto:759optional string type_name = 1
optional string type_version = 2
If not set, it looks for the type with type_name with default type_version.
optional options = 3
Specify List options. Currently supports: 1. Field to order the results. 2. Page size. If set, the request will first fetch all executions with specified `type_name` and `type_version`, then order by a specifield field finally find the correct page and return #Executions of the page size. Higher-level APIs may only use the functionalies partially. Please reference the API documentation for the API behaviors.
optional transaction_options = 4
Options regarding transactions.
message
metadata_store_service.proto:778repeated executions = 1
optional string next_page_token = 2
Token to use to retrieve next page of results if list options are used in the request.
Gets all the contexts of a given type.
message
metadata_store_service.proto:916optional string type_name = 1
optional options = 2
Specify options. Currently supports: 1. Field to order the results. 2. Page size. If set, the request will first fetch all contexts with specified `type_name` and `type_version`, then order by a specifield field finally find the correct page and return #Contexts of the page size. Higher-level APIs may only use the functionalies partially. Please reference the API documentation for the API behaviors.
optional string type_version = 3
If not set, it looks for the type with type_name and options with default type_version.
optional transaction_options = 4
Options regarding transactions.
message
metadata_store_service.proto:936repeated contexts = 1
optional string next_page_token = 2
Token to use to retrieve next page of results if list options are used in the request.
Gets the artifact of the given type and artifact name.
message
metadata_store_service.proto:564optional string type_name = 1
optional string type_version = 3
If not set, it looks for the type with type_name and artifact_name with default type_version.
optional string artifact_name = 2
optional transaction_options = 4
Options regarding transactions.
message
metadata_store_service.proto:574optional artifact = 1
Gets the execution of the given type and execution name.
message
metadata_store_service.proto:785optional string type_name = 1
optional string type_version = 3
If not set, it looks for the type with type_name and execution_name with default type_version.
optional string execution_name = 2
optional transaction_options = 4
Options regarding transactions.
message
metadata_store_service.proto:795optional execution = 1
Gets the context of the given type and context name.
message
metadata_store_service.proto:943optional string type_name = 1
optional string type_version = 3
If not set, it looks for the type with type_name and context_name with default type_version.
optional string context_name = 2
optional transaction_options = 4
Options regarding transactions.
message
metadata_store_service.proto:953optional context = 1
Gets all the artifacts with matching uris.
message
metadata_store_service.proto:626repeated string uris = 2
A list of artifact uris to retrieve.
optional transaction_options = 3
Options regarding transactions.
message
metadata_store_service.proto:635repeated artifacts = 1
Gets all events with matching execution ids.
message
metadata_store_service.proto:827Gets all events with matching execution ids.
repeated int64 execution_ids = 1
optional transaction_options = 2
Options regarding transactions.
message
metadata_store_service.proto:833repeated events = 1
Gets all events with matching artifact ids.
message
metadata_store_service.proto:837repeated int64 artifact_ids = 1
optional transaction_options = 2
Options regarding transactions.
message
metadata_store_service.proto:843repeated events = 1
Gets all the artifacts with matching external ids.
message
metadata_store_service.proto:699repeated string external_ids = 1
optional transaction_options = 2
Options regarding transactions.
message
metadata_store_service.proto:705repeated artifacts = 1
Gets all the artifacts with matching external ids.
message
metadata_store_service.proto:709repeated string external_ids = 1
optional transaction_options = 2
Options regarding transactions.
message
metadata_store_service.proto:715repeated executions = 1
Gets all the artifacts with matching external ids.
message
metadata_store_service.proto:719repeated string external_ids = 1
optional transaction_options = 2
Options regarding transactions.
message
metadata_store_service.proto:725repeated contexts = 1
Gets all the artifacts with matching external ids.
message
metadata_store_service.proto:729repeated string external_ids = 1
optional transaction_options = 2
Options regarding transactions.
message
metadata_store_service.proto:735repeated artifact_types = 1
Gets all the artifacts with matching external ids.
message
metadata_store_service.proto:739repeated string external_ids = 1
optional transaction_options = 2
Options regarding transactions.
message
metadata_store_service.proto:745repeated execution_types = 1
Gets all the artifacts with matching external ids.
message
metadata_store_service.proto:749repeated string external_ids = 1
optional transaction_options = 2
Options regarding transactions.
message
metadata_store_service.proto:755repeated context_types = 1
Gets all context that an artifact is attributed to.
message
metadata_store_service.proto:970optional int64 artifact_id = 1
optional transaction_options = 2
Options regarding transactions.
message
metadata_store_service.proto:976repeated contexts = 1
Gets all context that an execution is associated with.
message
metadata_store_service.proto:980optional int64 execution_id = 1
optional transaction_options = 2
Options regarding transactions.
message
metadata_store_service.proto:986repeated contexts = 1
Gets all parent contexts that a context is related.
message
metadata_store_service.proto:990optional int64 context_id = 1
optional transaction_options = 2
Options regarding transactions.
message
metadata_store_service.proto:996repeated contexts = 1
Gets all children contexts that a context is related.
optional int64 context_id = 1
optional transaction_options = 2
Options regarding transactions.
repeated contexts = 1
Batch getting all the parent contexts that a list of contexts are related.
repeated int64 context_ids = 1
optional transaction_options = 2
Options regarding transactions.
map<int64, > contexts = 2
Batch getting all the children contexts that a list of contexts are related.
repeated int64 context_ids = 1
optional transaction_options = 2
Options regarding transactions.
map<int64, > contexts = 2
Gets all direct artifacts that a context attributes to.
optional int64 context_id = 1
optional options = 2
Specify List options. Currently supports: 1. Field to order the results. 2. Page size.
optional transaction_options = 3
Options regarding transactions.
repeated artifacts = 1
optional string next_page_token = 2
Token to use to retrieve next page of results if list options are used in the request.
Gets all direct executions that a context associates with.
optional int64 context_id = 1
optional options = 2
Specify List options. Currently supports: 1. Field to order the results. 2. Page size.
optional transaction_options = 3
Options regarding transactions.
repeated executions = 1
optional string next_page_token = 2
Token to use to retrieve next page of results if list options are used in the request.
optional transaction_options = 3
Options regarding transactions.
Deprecated: GetLineageGraph API is deprecated, please refer to GetLineageSubgraph API as the alternative. The transaction performs a constrained transitive closure and returns a lineage subgraph satisfying the conditions and constraints specified in the GetLineageGraphRequest.
Deprecated: GetLineageGraph API is deprecated, please refer to GetLineageSubgraph API as the alternative. A lineage query request to specify the query nodes of interest and the boundary conditions for pruning the returned graph.
optional options = 1
optional transaction_options = 2
Options regarding transactions.
Deprecated: GetLineageGraph API is deprecated, please refer to GetLineageSubgraph API as the alternative. A connected lineage `subgraph` about the MLMD nodes derived from LineageGraphRequest.query_conditions.
optional subgraph = 1
Gets a lineage subgraph by performing graph traversal from a list of interested nodes. A lineage subgraph without node details (e.g., external_id, properties) will be returned.
optional lineage_subgraph_query_options = 1
Query options for lineage graph tracing from a list of interested nodes. A lineage subgraph without node details (e.g., external_id, properties) will be returned. Please refer to `LineageSubgraphQueryOptions` for more details.
optional read_mask = 3
`read_mask` contains user specified paths of fields that should be included in the returned lineage subgraph. Supported field paths are: 'artifacts', 'executions', 'contexts', 'artifact_types', 'execution_types', 'context_types', 'events', 'associations' and 'attributions'. If 'artifacts', 'executions', or 'contexts' is specified in `read_mask`, the dehydrated nodes will be included. Note: A dehydrated node means a node containing only its id and no other information. User should call GetNodesByID or other APIs to get node details later on. If 'artifact_types', 'execution_types', or 'context_types' is specified in `read_mask`, all the node types will be included. If 'events', 'associations', or 'attributions' is specified in `read_mask`, the corresponding edges will be included. If `read_mask` is not set, the API will return all the fields in the returned graph. Note: Only paths of fields in LineageGraph message are supported. Paths of fields in the submessage, such as "artifacts.id", "contexts.name" are not acknowledged.
optional transaction_options = 2
optional lineage_subgraph = 1
A lineage subgraph of MLMD nodes and relations retrieved from lineage graph tracing.
message
metadata_store.proto:577Every ArtifactStruct is a member of this type.
Used in:
(message has no fields)
message
metadata_store.proto:45Used in: , , , , , , , , , , ,
optional int64 id = 1
Output only. The unique server generated id of the artifact.
optional string name = 7
The client provided name of the artifact. This field is optional. If set, it must be unique among all the artifacts of the same artifact type within a database instance and cannot be changed once set.
optional int64 type_id = 2
The id of an ArtifactType. This needs to be specified when an artifact is created, and it cannot be changed.
optional string type = 8
Output only. The name of an ArtifactType.
optional string uri = 3
The uniform resource identifier of the physical artifact. May be empty if there is no physical artifact.
optional string external_id = 11
The external id that come from the clients’ system. This field is optional. If set, it must be unique among all artifacts within a database instance.
map<string, > properties = 4
Properties of the artifact. Properties must be specified in the ArtifactType.
map<string, > custom_properties = 5
User provided custom properties which are not defined by its type.
optional Artifact.State state = 6
The state of the artifact known to the system.
optional int64 create_time_since_epoch = 9
Output only. Create time of the artifact in millisecond since epoch.
optional int64 last_update_time_since_epoch = 10
Output only. Last update time of the artifact since epoch in millisecond since epoch.
optional system_metadata = 12
Output only.
Used in:
UNKNOWN = 0
PENDING = 1
A state indicating that the artifact may exist.
LIVE = 2
A state indicating that the artifact should exist, unless something external to the system deletes it.
MARKED_FOR_DELETION = 3
A state indicating that the artifact should be deleted.
DELETED = 4
A state indicating that the artifact has been deleted.
ABANDONED = 5
A state indicating that the artifact has been abandoned, which may be due to a failed or cancelled execution.
REFERENCE = 6
A state indicating that the artifact is a reference artifact. At execution start time, the orchestrator produces an output artifact for each output key with state PENDING. However, for an intermediate artifact, this first artifact's state will be REFERENCE. Intermediate artifacts emitted during a component's execution will copy the REFERENCE artifact's attributes. At the end of an execution, the artifact state should remain REFERENCE instead of being changed to LIVE.
message
metadata_store_service.proto:24An artifact and type pair. Part of an artifact struct.
Used in:
optional artifact = 1
optional type = 2
message
metadata_store_service.proto:44An artifact struct represents the input or output of an Execution. See the more specific types referenced in the message for more details.
Used in: ,
oneof value
Note: an artifact struct may be empty to indicate "None" or null.
artifact = 1
map = 2
list = 3
message
metadata_store_service.proto:37An artifact struct that is a list.
Used in:
repeated elements = 1
Can be represented as a JSON list of artifact structs.
message
metadata_store_service.proto:30A dictionary of artifact structs. Can represent a dictionary.
Used in:
map<string, > properties = 1
An artifact struct that is a dictionary. Can be represented as a JSON dictionary of artifact structs.
message
metadata_store.proto:497The list of ArtifactStruct is EXPERIMENTAL and not in use yet. The type of an ArtifactStruct. An artifact struct type represents an infinite set of artifact structs. It can specify the input or output type of an ExecutionType. See the more specific types referenced in the message for more details.
Used in: , , , , ,
oneof kind
simple = 1
Matches exactly this type.
union_type = 2
intersection = 3
list = 4
none = 5
any = 6
tuple = 7
dict = 8
message
metadata_store.proto:118Used in: , , , , , , , , , ,
optional int64 id = 1
The id of the type. 1-1 relationship between type names and IDs.
optional string name = 2
The name of the type. It must be unique among ArtifactTypes within a database instance.
optional string version = 4
An optional version of the type. An empty string is treated as unset.
optional string description = 5
An optional description about the type.
optional string external_id = 7
The external id that come from the clients’ system. This field is optional. If set, it must be unique among all artifact types within a database instance.
map<string, PropertyType> properties = 3
The schema of the type. Properties are always optional in the artifact. Properties of an artifact type can be expanded but not contracted (i.e., you can add columns but not remove them).
optional ArtifactType.SystemDefinedBaseType base_type = 6
An optional system defined base_type expressing the intent of the current type. This field is useful for the tool builders to utilize the stored MLMD information, e.g., `MyModel` ArtifactType could set base_type = MODEL.
An enum of system-defined artifact types.
Used in:
UNSET = 0
DATASET = 1
MODEL = 2
METRICS = 3
STATISTICS = 4
message
metadata_store.proto:456the Association edges between Context and Execution instances.
Used in: ,
optional int64 execution_id = 1
optional int64 context_id = 2
optional system_metadata = 3
Output only.
message
metadata_store.proto:447the Attribution edges between Context and Artifact instances.
Used in: ,
optional int64 artifact_id = 1
optional int64 context_id = 2
optional system_metadata = 3
Output only.
message
metadata_store.proto:790Used in:
oneof config
Configuration for a new connection.
fake_database = 1
mysql = 2
sqlite = 3
postgresql = 5
PostgreSQL database connection config.
optional retry_options = 4
Options for overwriting the default retry setting when MLMD transactions returning Aborted error. The setting is currently available for python client library only. TODO(b/154862807) set the setting in transaction executor.
message
metadata_store.proto:419Used in: , , , , , , , , , , , , , ,
optional int64 id = 1
Output Only. The unique server generated id of the context.
optional string name = 3
The client provided name of the context. It must be unique within a database instance.
optional int64 type_id = 2
The id of a ContextType. This needs to be specified when a context is created, and it cannot be changed.
optional string type = 6
Output only. The name of a ContextType.
optional string external_id = 9
The external id that come from the clients’ system. This field is optional. If set, it must be unique among all contexts within a virtual database.
map<string, > properties = 4
Values of the properties, which must be specified in the ContextType.
map<string, > custom_properties = 5
User provided custom properties which are not defined by its type.
optional int64 create_time_since_epoch = 7
Output only. Create time of the context in millisecond since epoch.
optional int64 last_update_time_since_epoch = 8
Output only. Last update time of the context in millisecond since epoch.
optional system_metadata = 10
Output only system metadata.
message
metadata_store.proto:387Used in: , , , , , , ,
optional int64 id = 1
The id of the type. 1-1 relationship between type names and IDs.
optional string name = 2
The name of the type, e.g., Pipeline, Task, Session, User, etc. It must be unique among ContextTypes within a database instance.
optional string version = 4
An optional version of the type. An empty string is treated as unset.
optional string description = 5
An optional description about the type.
optional string external_id = 7
The external id that come from the clients’ system. This field is optional. If set, it must be unique among all context types within a database instance.
map<string, PropertyType> properties = 3
The schema of the type, e.g., name: string, owner: string Properties are always optional in the context. Properties of an context type can be expanded but not contracted (i.e., you can add columns but not remove them).
optional ContextType.SystemDefinedBaseType base_type = 6
An optional system defined base_type expressing the intent of the current context type. *NOTE: currently there are no system Context types defined, and the field is not used for ContextType.
An enum of system-defined context types.
Used in:
UNSET = 0
message
metadata_store.proto:588A artifact struct type that represents a record or struct-like dictionary. ArtifactStruct would be map (i.e. ArtifactStructMap)
Used in:
map<string, > properties = 1
Underlying properties for the type.
optional bool none_type_not_required = 2
If true, then if properties["foo"] can be None, then that key is not required.
optional extra_properties_type = 3
Extra keys are allowed that are not specified in properties. These keys must have the type specified below. If this is not specified, then extra properties are not allowed.
message
metadata_store.proto:238An event represents a relationship between an artifact and an execution. There are different kinds of events, relating to both input and output, as well as how they are used by the mlmd powered system. For example, the DECLARED_INPUT and DECLARED_OUTPUT events are part of the signature of an execution. For example, consider: my_result = my_execution({"data":[3,7],"schema":8}) Where 3, 7, and 8 are artifact_ids, Assuming execution_id of my_execution is 12 and artifact_id of my_result is 15, the events are: { artifact_id:3, execution_id: 12, type:DECLARED_INPUT, path:{step:[{"key":"data"},{"index":0}]} } { artifact_id:7, execution_id: 12, type:DECLARED_INPUT, path:{step:[{"key":"data"},{"index":1}]} } { artifact_id:8, execution_id: 12, type:DECLARED_INPUT, path:{step:[{"key":"schema"}]} } { artifact_id:15, execution_id: 12, type:DECLARED_OUTPUT, path:{step:[{"key":"my_result"}]} } Other event types include INPUT/OUTPUT, INTERNAL_INPUT/_OUTPUT and PENDING_OUTPUT: * The INPUT/OUTPUT is an event that actually reads/writes an artifact by an execution. The input/output artifacts may not declared in the signature, For example, the trainer may output multiple caches of the parameters (as an OUTPUT), then finally write the SavedModel as a DECLARED_OUTPUT. * The INTERNAL_INPUT/_OUTPUT are event types which are only meaningful to an orchestration system to keep track of the details for later debugging. For example, a fork happened conditioning on an artifact, then an execution is triggered, such fork implementing may need to log the read and write of artifacts and may not be worth displaying to the users. For instance, in the above example, my_result = my_execution({"data":[3,7],"schema":8}) there is another execution (id: 15), which represents a `garbage_collection` step in an orchestration system gc_result = garbage_collection(my_result) that cleans `my_result` if needed. The details should be invisible to the end users and lineage tracking. The orchestrator can emit following events: { artifact_id: 15, execution_id: 15, type:INTERNAL_INPUT, } { artifact_id:16, // New artifact containing the GC job result. execution_id: 15, type:INTERNAL_OUTPUT, path:{step:[{"key":"gc_result"}]} } * The PENDING_OUTPUT event is used to indicate that an artifact is tentatively associated with an active execution which has not yet been finalized. For example, an orchestration system can register output artifacts of a running execution with PENDING_OUTPUT events to indicate the output artifacts the execution is expected to produce. When the execution is finished, the final set of output artifacts can be associated with the exeution using OUTPUT events, and any unused artifacts which were previously registered with PENDING_OUTPUT events can be updated to set their Artifact.State to ABANDONED. Events are unique of the same (artifact_id, execution_id, type) combination within a metadata store.
Used in: , , , , ,
optional int64 artifact_id = 1
The artifact id is required for an event, and should refer to an existing artifact.
optional int64 execution_id = 2
The execution_id is required for an event, and should refer to an existing execution.
optional path = 3
The path in an artifact struct, or the name of an artifact.
optional Event.Type type = 4
The type of an event.
optional int64 milliseconds_since_epoch = 5
Time the event occurred Epoch is Jan 1, 1970, UTC
optional system_metadata = 6
Output only.
message
metadata_store.proto:241A simple path (e.g. {step{key:"foo"}}) can name an artifact in the context of an execution.
Used in:
repeated steps = 1
A simple path (e.g. {step{key:"foo"}}) can name an artifact in the context of an execution.
message
metadata_store.proto:242Used in:
oneof value
int64 index = 1
string key = 2
Events distinguish between an artifact that is written by the execution (possibly as a cache), versus artifacts that are part of the declared output of the Execution. For more information on what DECLARED_ means, see the comment on the message.
Used in:
UNKNOWN = 0
DECLARED_OUTPUT = 1
A declared output of the execution.
DECLARED_INPUT = 2
A declared input of the execution.
INPUT = 3
An input of the execution.
OUTPUT = 4
An output of the execution.
INTERNAL_INPUT = 5
An internal input of the execution.
INTERNAL_OUTPUT = 6
An internal output of the execution.
PENDING_OUTPUT = 7
A pending output of the execution.
message
metadata_store.proto:284Used in: , , , , , , , , ,
optional int64 id = 1
Output only. The unique server generated id of the execution.
optional string name = 6
The client provided name of the execution. This field is optional. If set, it must be unique among all the executions of the same execution type within a database instance and cannot be changed once set.
optional int64 type_id = 2
The id of an ExecutionType. This needs to be specified when an execution is created, and it cannot be changed. The id of an ExecutionType.
optional string type = 7
Output only. The name of an ExecutionType.
optional string external_id = 10
The external id that come from the clients’ system. This field is optional. If set, it must be unique among all executions within a database instance.
optional Execution.State last_known_state = 3
The last known state of an execution in the system.
map<string, > properties = 4
Properties of the Execution. Properties must be specified in the ExecutionType.
map<string, > custom_properties = 5
User provided custom properties which are not defined by its type.
optional int64 create_time_since_epoch = 8
Output only. Create time of the execution in millisecond since epoch.
optional int64 last_update_time_since_epoch = 9
Output only. Last update time of the execution in millisecond since epoch.
optional system_metadata = 11
Output only.
The state of the Execution. The state transitions are NEW -> RUNNING -> COMPLETE | CACHED | FAILED | CANCELED CACHED means the execution is skipped due to cached results. CANCELED means the execution is skipped due to precondition not met. It is different from CACHED in that a CANCELED execution will not have any event associated with it. It is different from FAILED in that there is no unexpected error happened and it is regarded as a normal state.
Used in:
UNKNOWN = 0
NEW = 1
RUNNING = 2
COMPLETE = 3
FAILED = 4
CACHED = 5
CANCELED = 6
message
metadata_store.proto:332Used in: , , , , , , ,
optional int64 id = 1
The id of the type. 1-1 relationship between type names and IDs.
optional string name = 2
The name of the type. It must be unique among ExecutionTypes within a database instance.
optional string version = 6
An optional version of the type. An empty string is treated as unset.
optional string description = 7
An optional description about the type.
optional string external_id = 9
The external id that come from the clients’ system. This field is optional. If set, it must be unique among all execution types within a database instance.
map<string, PropertyType> properties = 3
The schema of the type. Properties are always optional in the execution.
optional input_type = 4
The ArtifactStructType of the input. For example: { "dict":{ "properties":{ "schema":{ "union_type":{ "none":{}, "simple":{...schema type...} }, }, "data":{ "simple":{...data_type...} } } } } That would be an optional schema field with a required data field.
optional output_type = 5
The ArtifactStructType of the output. For example {"simple":{...stats gen output type...}}
optional ExecutionType.SystemDefinedBaseType base_type = 8
An optional system defined base_type expressing the intent of the current type. This field is useful for the tool builders to utilize the stored MLMD information, e.g., `MyTrainer` ExecutionType could set base_type = TRAIN.
An enum of system-defined execution types.
Used in:
UNSET = 0
TRAIN = 1
TRANSFORM = 2
PROCESS = 3
EVALUATE = 4
DEPLOY = 5
message
metadata_store.proto:605Configuration for a "fake" database. This database is an in-memory SQLite database that lives only as long as the associated object lives.
Used in:
(message has no fields)
Used in:
repeated children_contexts = 1
Used in:
repeated parent_contexts = 1
message
metadata_store.proto:811A list of supported GRPC arguments defined in: https://grpc.github.io/grpc/core/group__grpc__arg__keys.html
Used in:
optional int64 max_receive_message_length = 1
Maximum message length in bytes per response that the channel can receive.
optional int64 http2_max_ping_strikes = 2
Maximum misbehaving pings the server can bear before sending goaway and closing the transport? (0 indicates infinite number of misbehaving pings)
message
metadata_store.proto:541A member of this type must satisfy all constraints. This primarily useful not as an end-user type, but something calculated as an intermediate type in the system. For example, suppose you have a method: def infer_my_input_type(a): # try to infer the input type of this method. use_in_method_x(a) # with input type x_input use_in_method_y(a) # with input type y_input Given this information, you know that infer_my_input_type has type {"intersection":{"constraints":[x_input, y_input]}}. IntersectionArtifactStructType intersection_type = {"constraints":[ {"dict":{"properties":{"schema":{"any":{}}}, "extra_properties":{"any":{}}}}, {"dict":{"properties":{"data":{"any":{}}}, "extra_properties":{"any":{}}}}]} Since the first constraint requires the dictionary to have a schema property, and the second constraint requires it to have a data property, this is equivalent to: ArtifactStructType other_type = {"dict":{"properties":{"schema":{"any":{}},"data":{"any":{}}}}, "extra_properties":{"any":{}}}
Used in:
repeated constraints = 1
message
metadata_store.proto:476A self-contained provenance (sub)graph representation consists of MLMD nodes and their relationships. It is used to represent the query results from the persistent backend (e.g., lineage about a node, reachability of two nodes).
Used in: ,
repeated artifact_types = 1
extracted types
repeated execution_types = 2
repeated context_types = 3
repeated artifacts = 4
extracted nodes
repeated executions = 5
repeated contexts = 6
repeated events = 7
extracted edges
repeated attributions = 8
repeated associations = 9
repeated parent_contexts = 10
message
metadata_store.proto:1081Deprecated: GetLineageGraph API is deprecated, please refer to GetLineageSubgraph API as the alternative. The query options for `get_lineage_graph` operation. `query_nodes` is a list of nodes of interest. Currently only artifacts are supported as `query_nodes`. `stop_conditions` defines the filtering rules when querying a lineage graph. `max_node_size` defines the total number of artifacts and executions returned in the subgraph.
Used in:
oneof query_nodes
A query to specify the nodes of interest. `ListOperationOptions.max_result_size` sets the maximum number of nodes to begin with the graph search. TODO(b/178491112) Support query_nodes for Executions.
artifacts_options = 1
optional stop_conditions = 2
A constraint option to define the filtering rules when querying a lineage graph.
optional int64 max_node_size = 3
Maximum total number of artifacts and executions in the whole returned lineage graph. If set to 0 or below, all related nodes will be returned without any number limitation. The number counts toward Artifacts and Executions. Nothing else considered. NOTE: There is no pagination supported.
message
metadata_store.proto:1093Filtering conditions for retrieving the lineage graph.
Used in:
optional int64 max_num_hops = 1
The maximum number of hops from the `query_nodes` to traverse. A hop is defined as a jump to the next node following the path of node -> event -> next_node. For example, in the lineage graph a_1 -> e_1 -> a_2: a_2 is 2 hops away from a_1, and e_1 is 1 hop away from a_1. `max_num_hops` should be non-negative. When its value is set to 0, only the `query_nodes` are returned.
optional string boundary_artifacts = 2
Filtering conditions for retrieving the lineage graph. Please refer to `ListOperationOptions.filter_query` for the syntax. If set, the `boundary_artifacts` defines which artifacts to keep in the returned lineage graph during the graph search. Artifacts that do not satisfy the `boundary_artifacts` are filtered out, and the subgraphs starting at them will be pruned. If not set, no artifacts will be filtered out. Taking the following lineage graph as example: (`a` represents an Artifact, `e` represents an Execution, each arrow represents a hop.) a_0 a_1 a_3 | \ / \ \/ \/ \/ \/ e_0 e_1 e_3 / \ \/ \/ a_2 a_4 a_5 \ / \/ \/ e_2 To query all the upstream and downstream nodes 3 hops away from a_4, while excluding the upstream subgraph starting at a_3, then `stop_conditions` can be set as: { max_num_hops: 3 boundary_artifacts: 'id != 3' } With the `stop_conditions`, {a_3, e_1, a_1, a_0, e_0} will be filtered out. The returned lineage graph looks like: e_3 / \ \/ \/ a_2 a_4 a_5 \ / \/ \/ e_2
optional string boundary_executions = 3
If set, the `boundary_executions` defines which executions to keep in the returned lineage graph during the graph search. Executions that do not satisfy the `boundary_executions` are filtered out and the subgraphs starting at them will be pruned. If not set, no executions will be filtered out. In the example above, to query for all the upstream and downstream nodes 3 hops away from a_4, while excluding the upstream subgraph and the downstream subgraph starting at e_3, then `stop_conditions` can be set as { max_num_hops: 3 boundary_executions: 'id != 3' } With the `stop_conditions`, {e_3, a_5, a_3, e_1, a_1, a_0, e_0} will be filtered out. The returned lineage graph looks like: a_2 a_4 \ / \/ \/ e_2 However, for the following graph: a_0 a_1 a_3 | \ / \ \/ \/ \/ \/ e_0 e_1 e_3 \ / \ \/ \/ \/ a_2 a_4 a_5 \ / \/ \/ e_2 With the same `stop_conditions`, only {e_3, a_5, a_0, e_0} will be filtered out. The returned lineage graph looks like: a_1 a_3 \ / \/ \/ e_1 \ \/ a_2 a_4 \ / \/ \/ e_2
message
metadata_store.proto:1203The query options for lineage graph tracing from a list of interested nodes.
Used in:
oneof starting_nodes
starting_artifacts = 1
starting_executions = 2
optional int64 max_num_hops = 3
The maximum number of hops from the `starting_nodes` to traverse. A hop is defined as a jump to the next node following the path of node -> event -> next_node. For example, in the lineage graph a_1 -> e_1 -> a_2: a_2 is 2 hops away from a_1, and e_1 is 1 hop away from a_1. `max_num_hops` should be non-negative. When its value is set to 0, only the `starting_nodes` are returned.
optional LineageSubgraphQueryOptions.Direction direction = 4
The direction of lineage graph tracing, which means the direction of all hops in the tracing. A DOWNSTREAM hop means an expansion following the path of execution -> output_event -> artifact or artifact -> input_event -> execution An UPSTREAM hop means an expansion following the path of execution -> input_event -> artifact or artifact -> output_event -> execution Please refer to `Direction` for more details.
optional ending_artifacts = 5
If set, `ending_artifacts` defines the terminal artifacts within `max_num_hops` of the traversed subgraph. No further expansion will occur at these terminal artifact nodes. If not set, expansion will continue until the traversal reaches `max_num_hops`. Taking the following lineage graph as example: (`a` represents an Artifact, `e` represents an Execution, each arrow represents a hop.) a_0 a_1 a_3 | \ / \ \/ \/ \/ \/ e_0 e_1 e_3 / \ \/ \/ a_2 a_4 a_5 \ / \/ \/ e_2 To query all the upstream and downstream nodes 3 hops away from a_4, and end traversal if a_3 is met, `ending_artifacts` can be set as: { filter_query: 'id = 3' include_ending_nodes: false } The returned lineage graph looks like: e_3 / \ \/ \/ a_2 a_4 a_5 \ / \/ \/ e_2 `ending_artifacts` and `ending_executions` can be combined in the request. Taking the same graph as example, if we combine `ending_artifacts` defined above with `ending_executions` as: { filter_query: 'id = 2' include_ending_nodes: true } The returned lineage graph looks like: e_3 / \ \/ \/ a_4 a_5 / \/ e_2
optional ending_executions = 6
If set, `ending_executions` defines the terminal executions within `max_num_hops` of the traversed subgraph. No further expansion will occur at these terminal execution nodes. If not set, expansion will continue until the traversal reaches `max_num_hops`.
Used in:
DIRECTION_UNSPECIFIED = 0
Direction is by defult DIRECTION_UNSPECIFIED, which is equivalent to BIDIRECTIONAL.
UPSTREAM = 1
Indicates tracing the lineage graph by hops in upstream direction.
DOWNSTREAM = 2
Indicates tracing the lineage graph by hops in downstream direction.
BIDIRECTIONAL = 3
Indicates tracing the lineage graph in both directions.
message
metadata_store.proto:1250`ending_nodes` is a list of nodes that end expanding the graph.
Used in:
optional string filter_query = 1
`filter_query` is a boolean expression in SQL syntax that is used to specify the conditions on ending nodes. Please refer to ListOperationOptions.filter_query for more details.
optional bool include_ending_nodes = 2
If true, include the ending nodes defined by the filter query, as well as edges connected from the traversed subgraph to them. If false, do not include the nodes and edges connected to them.
message
metadata_store.proto:1206`starting_nodes` is a list of nodes of interest to start graph tracing. NOTE: The maximum number of starting nodes is 100 at most.
Used in:
optional string filter_query = 1
`filter_query` is a boolean expression in SQL syntax that is used to specify the conditions on starting nodes. Please refer to ListOperationOptions.filter_query for more details.
message
metadata_store.proto:546Represents an ArtifactStruct list type with homogeneous elements.
Used in:
optional element = 1
Every entry in the list must be of this type. Note: if this type is Any, then the list can have arbitrary elements.
message
metadata_store.proto:1036Encapsulates information to identify the next page of resources in ListOperation.
optional int64 id_offset = 1
Id offset within the resultset to start next page. Id offset is returned as Id is the unique field used to break ties for fields that might have duplicate entries, e.g. there could be two resources with same create_time. In such cases to break the tie in ordering, id offset is used. This field is currently only set whe order_by field is CREATE_TIME.
optional int64 field_offset = 2
Offset value of the order by field. If ID is used this value is same as id_offset.
optional set_options = 3
Options set in the first call to ListOperation. This ensures that if next_page_token is set by the caller then ListPipelineJobs API will always use options set in the first call.
repeated int64 listed_ids = 4
List of ids that have the same order_by field values. This is used to ensure List Operation does not return duplicate entries for nodes that have the same order_by field value. This field is currently only set whe order_by field is LAST_UPDATE_TIME.
message
metadata_store.proto:880ListOperationOptions represents the set of options and predicates to be used for List operations on Artifacts, Executions and Contexts.
Used in: , , , , , , , , ,
optional int32 max_result_size = 1
Max number of resources to return in the result. A value of zero or less results in a InvalidArgumentError. The API implementation also enforces an upper-bound of 100, and picks the minimum between this value and the one specified here.
optional order_by_field = 2
Ordering field.
optional string next_page_token = 3
Identifies the next page of results.
optional string filter_query = 4
A boolean expression in SQL syntax that is used to specify the conditions on node attributes and directly connected assets. In the current implementation, filtering Artifact/Execution/Context with the following attributes and neighborhood is supported: Attributes: id:int64, type_id:int64, type:string, uri:string, name: string, external_id: string, create_time_since_epoch:int64, last_update_time_since_epoch:int64 state:ENUM (Artifact only) last_known_state:ENUM (Execution only) Neighborhood - Properties and Custom Properties (for all node types): syntax: properties.$name ($name is the property name) custom_properties.$name ($name is the custom property name) attributes: the following attributes can be used int_value: int64, double_value: double, string_value: string bool_value: bool - Context (for Artifact and Execution): syntax: contexts_$alias ($alias can be [0-9A-Za-z_]) attributes: the following attributes can be used id:int64, name:string, type:string, create_time_since_epoch:int64, last_update_time_since_epoch: int64 - Parent and Child Contexts (for Contexts): syntax: parent_contexts_$alias( $alias can be [0-9A-Za-z_] child_contexts_$alias( $alias can be [0-9A-Za-z_] attributes: the following attributes can be used id:int64, name: string, type:string - Event (for Artifact and Execution) syntax: events_$alias ($alias can be [0-9A-Za-z_]) attributes: the following attributes can be used artifact_id: int64(Execution only), execution_id: int64(Artifact only), type: ENUM, milliseconds_since_epoch: int64 Examples: a) to filter nodes attributes: - id != 1 - id IN (1, 3) - type_id = 5 - type = 'my_type_name' - name = 'foo' - type = 'bar' AND name LIKE 'foo%' - external_id = 'my_external_id' - NOT(create_time_since_epoch < 1 OR last_update_time_since_epoch < 1) b) to filter artifacts' uri - uri = 'exact_path_string' - uri LIKE 'path_like_this%' - uri IS NOT NULL c) to filter artifact's state or execution's last_known_state - state = LIVE - state IS NULL - state IN (PENDING, LIVE) - last_known_state = RUNNING - last_known_state != RUNNING - last_known_state NOT IN (FAILED, CANCELED) d) to filter nodes having a specific context, artifact, or execution - contexts_a.id = 5 - contexts_a.type = 'RunContext' - contexts_a.name = 'my_run' - contexts_a.create_time_since_epoch = 1626761453 - contexts_a.last_update_time_since_epoch = 1626761453 To filter nodes with conditions on multiple contexts: - contexts_a.name = 'my_run' AND contexts_b.name = 'my_pipeline' To filter context with artifacts: - artifacts_a.id = 5 - artifacts_a.type = 'Dataset' - artifacts_a.name = 'my_dataset' - artifacts_a.uri = 'exact_path_string' - artifacts_a.state = LIVE - artifacts_a.state IN (PENDING, LIVE) - artifacts_a.external_id = "my_external_id" - artifacts_a.create_time_since_epoch = 1626761453 - artifacts_a.last_update_time_since_epoch = 1626761453 To filter contexts with conditions on multiple artifacts: - artifacts_a.name = 'my_run' AND artifacts_b.name = 'my_pipeline' To filter context with executions: - executions_a.id = 5 - executions_a.type = 'Dataset' - executions_a.name = 'my_dataset' - executions_a.last_known_state = RUNNING . - executions_a.last_known_state IN (NEW, RUNNING) - executions_a.external_id = "my_external_id" - executions_a.create_time_since_epoch = 1626761453 - executions_a.last_update_time_since_epoch = 1626761453 To filter contexts with conditions on multiple executions: - executions_a.name = 'my_run' AND executions_b.name = 'my_pipeline' e) to filter nodes condition on their properties - properties.accuracy.double_value > 0.95 - custom_properties.my_param.string_value = "foo" If the name of the property or custom property includes characters other than [0-9A-Za-z_], then the name need to be backquoted, e.g., - properties.`my property`.int_value > 0 - custom_properties.`my:custom.property`.bool_value = true f) complex query to filter both node attributes and neighborhood - type = 'DataSet' AND (contexts_a.type = 'RunContext' AND contexts_a.name = 'my_run') AND (properties.span = 1 OR custom_properties.span = 1) g) to filter parent/child context - parent_contexts_a.id = 5 - child_contexts_a.type = 'RunContext' - parent_contexts_a.name = 'parent_context_1' h) to filter Artifacts on Events - events_0.execution_id = 1 - events_0.type = INPUT - events_0.milliseconds_since_epoch = 1 to filter Executions on Events - events_0.artifact_id = 1 - events_0.type IN (INPUT, INTERNAL_INPUT) - events_0.milliseconds_since_epoch = 1 TODO(b/145945460) Support filtering on event step fields.
message
metadata_store.proto:887Used in:
optional OrderByField.Field field = 1
Field to order.
optional bool is_asc = 2
Direction of ordering.
Supported fields for Ordering.
Used in:
FIELD_UNSPECIFIED = 0
CREATE_TIME = 1
LAST_UPDATE_TIME = 2
ID = 3
message
metadata_source.proto:56A config includes a set of SQL queries and the type of metadata source. It is used by MetadataAccessObject to init backend and issue queries. Next ID: 144
MetadataSourceType metadata_source_type = 1
the type of the metadata source
optional drop_type_table = 3
Drops the Type table.
optional create_type_table = 4
Creates the Type table.
optional check_type_table = 44
Checks the existence of the Type table.
optional insert_artifact_type = 54
Inserts an artifact type into the Type table. It has 3 parameters. $0 is the name $1 is the version $2 is the description
optional insert_execution_type = 55
Inserts an execution type into the Type table. It has 3 parameters. $0 is the type name $1 is the version $2 is the description $3 is the input_type serialized as JSON or null. $4 is the output_type serialized as JSON or null.
optional insert_context_type = 58
Inserts a context type into the Type table. It has 1 parameter. $0 is the type name $1 is the version $2 is the description
optional select_types_by_id = 128
Queries types by a list of type ids. It has 2 parameter. $0 is the type_ids $1 is the is_artifact_type
optional select_types_by_external_ids = 134
Queries types by a list of external ids. It has 2 parameters. $0 is the external_ids $1 is the type_kind
optional select_type_by_id = 6
Queries a type by its type id. It has 2 parameters. $0 is the type id $1 is the is_artifact_type
optional select_type_by_name = 20
Queries a type by its type name. It has 2 parameters. $0 is the type name $1 is the type_kind
optional select_type_by_name_and_version = 111
Queries a type by its type name and version. It has 3 parameters. $0 is the type name $1 is the type version $2 is the type_kind
optional select_types_by_names = 136
Queries types by a list of type names. It has 2 parameters. $0 is the type names $1 is the type kind
optional select_types_by_names_and_versions = 137
Queries types by a list of type name and version pairs. It has 2 parameters. $0 is the type name and version pairs $1 is the type kind
optional select_all_types = 57
Queries for all type instances. It has 1 parameter. $0 is the is_artifact_type
optional update_type = 138
Updates a type in the Type table. It has 2 parameters. $0 is the existing type id $1 is the external_id of the Type
optional drop_parent_type_table = 99
Drops the ParentType table.
optional create_parent_type_table = 100
Creates the ParentType table.
optional check_parent_type_table = 101
Checks the existence of the ParentType table.
optional insert_parent_type = 109
Inserts a parent type into the ParentType table. It has 2 parameters: $0 is the type_id $1 is the parent_type_id
optional select_parent_type_by_type_id = 110
Queries parent types from the ParentType table by type_id. It has 1 parameter. $0 is the type_id
optional drop_type_property_table = 7
Drops the TypeProperty table.
optional create_type_property_table = 8
Creates the TypeProperty table.
optional check_type_property_table = 45
Checks the existence of the TypeProperty table.
optional insert_type_property = 9
Inserts a property of a type into the TypeProperty table. It has 3 parameters. $0 is the type_id $1 is the name of the property $2 is the data_type of the property
optional select_property_by_type_id = 10
Queries properties of a type from the TypeProperty table by the type_id Returns a list of properties (name, data_type). It has 1 parameter. $0 is the type_id
optional select_last_insert_id = 11
Queries the last inserted id.
optional drop_artifact_table = 12
Drops the Artifact table.
optional create_artifact_table = 13
Creates the Artifact table.
optional check_artifact_table = 46
Checks the existence of the Artifact table.
optional insert_artifact = 14
Inserts an artifact into the Artifact table. It has 5 parameters. $0 is the type_id $1 is the uri of the Artifact $2 is the name of the Artifact $3 is the create_time_since_epoch of the Artifact $4 is the last_update_time_since_epoch of the Artifact
optional select_artifact_by_id = 15
Queries an artifact from the Artifact table by its id. It has 1 parameter. $0 is the artifact_id
optional select_artifact_by_type_id_and_name = 94
Queries an artifact from the Artifact table by its name and type id. It has 2 parameter. $0 is the type_id $1 is the name of the Artifact
optional select_artifacts_by_type_id = 52
Queries an artifact from the Artifact table by its type_id. It has 1 parameter. $0 is the artifact_type_id
optional select_artifacts_by_uri = 56
Queries an artifact from the Artifact table by its uri. It has 1 parameter. $0 is the uri
optional select_artifacts_by_external_ids = 130
Queries artifacts from the Artifact table by external_ids. It has 1 parameter. $0 is the external_ids
optional update_artifact = 21
Updates an artifact in the Artifact table. It has 4 parameters. $0 is the existing artifact id $1 is the type_id $2 is the uri of the Artifact $3 is the last_update_time_since_epoch of the Artifact
optional drop_artifact_property_table = 16
Drops the ArtifactProperty table.
optional create_artifact_property_table = 17
Creates the ArtifactProperty table.
optional check_artifact_property_table = 47
Checks the existence of the ArtifactProperty table.
optional insert_artifact_property = 18
Insert a property of an artifact from the ArtifactProperty table. It has 5 parameters. $0 is the property data type $1 is the artifact_id $2 is the name of the artifact property $3 is the flag to indicate whether it is a custom property $4 is the value of the property
optional select_artifact_property_by_artifact_id = 19
Queries properties of an artifact from the ArtifactProperty table by the artifact id. It has 1 parameter. $0 is the artifact_id
optional update_artifact_property = 22
Updates a property of an artifact in the ArtifactProperty table. It has 4 parameters. $0 is the property data type $1 is the value of the property $2 is the artifact_id $3 is the name of the artifact property
optional delete_artifact_property = 23
Deletes a property of an artifact. It has 2 parameters. $0 is the artifact_id $1 is the name of the artifact property
optional drop_execution_table = 24
Drops the Execution table.
optional create_execution_table = 25
Creates the Execution table.
optional check_execution_table = 48
Checks the existence of the Execution table.
optional insert_execution = 28
Inserts an execution into the Execution table. It has 4 parameter. $0 is the type_id $1 is the name of the execution $2 is the create_time_since_epoch of the execution $3 is the last_update_time_since_epoch of the execution
optional select_execution_by_id = 29
Queries an execution from the Execution table by its id. It has 1 parameter. $0 is the execution_id
optional select_execution_by_type_id_and_name = 95
Queries an execution from the Execution table by its name and type id. It has 2 parameters. $0 is the type_id $1 is the name
optional select_executions_by_type_id = 53
Queries an execution from the Execution table by its type_id. It has 1 parameter. $0 is the execution_type_id
optional select_executions_by_external_ids = 132
Queries executions from the Execution table by external_ids. It has 1 parameter. $0 is the external_ids
optional update_execution = 34
Updates an execution in the Execution table. It has 3 parameters. $0 is the existing execution id $1 is the type_id $2 is the last_update_time_since_epoch of the execution
optional drop_execution_property_table = 26
Drops the ExecutionProperty table.
optional create_execution_property_table = 27
Creates the ExecutionProperty table.
optional check_execution_property_table = 49
Checks the existence of the ExecutionProperty table.
optional insert_execution_property = 30
Insert a property of an execution from the ExecutionProperty table. It has 5 parameters. $0 is the property data type $1 is the execution_id $2 is the name of the execution property $3 is the flag to indicate whether it is a custom property $4 is the value of the property
optional select_execution_property_by_execution_id = 31
Queries properties of an execution from the ExecutionProperty table by the execution id. It has 1 parameter. $0 is the execution_id
optional update_execution_property = 32
Updates a property of an execution in the ExecutionProperty table. It has 4 parameters. $0 is the property data type $1 is the value of the property $2 is the execution_id $3 is the name of the execution property
optional delete_execution_property = 33
Deletes a property of an execution. It has 2 parameters. $0 is the execution_id $1 is the name of the execution property
optional drop_context_table = 67
Drops the Context table.
optional create_context_table = 68
Creates the Context table.
optional check_context_table = 69
Checks the existence of the Context table.
optional insert_context = 70
Inserts a context into the Context table. It has 4 parameters. $0 is the type_id $1 is the name of the Context $2 is the create_time_since_epoch of the Context $3 is the last_update_time_since_epoch of the Context
optional select_context_by_id = 71
Queries a context from the Context table by its id. It has 1 parameter. $0 is the context_id
optional select_contexts_by_type_id = 72
Queries a context from the Context table by its type_id. It has 1 parameter. $0 is the context_type_id
optional select_context_by_type_id_and_name = 93
Queries a context from the Context table by its type_id and name. It has 2 parameters. $0 is the context_type_id $1 is the context_name
optional select_contexts_by_external_ids = 133
Queries contexts from the Context table by external_ids. It has 1 parameter. $0 is the external_ids
optional update_context = 73
Updates a context in the Context table. It has 4 parameters. $0 is the existing context id $1 is the type_id $2 is the name of the Context $3 is the last_update_time_since_epoch of the Context
optional drop_context_property_table = 74
Drops the ContextProperty table.
optional create_context_property_table = 75
Creates the ContextProperty table.
optional check_context_property_table = 76
Checks the existence of the ContextProperty table.
optional insert_context_property = 77
Insert a property of a context from the ContextProperty table. It has 5 parameters. $0 is the property data type $1 is the context_id $2 is the name of the context property $3 is the flag to indicate whether it is a custom property $4 is the value of the property
optional select_context_property_by_context_id = 78
Queries properties of a context from the ContextProperty table by the context id. It has 1 parameter. $0 is the context_id
optional update_context_property = 79
Updates a property of a context in the ContextProperty table. It has 4 parameters. $0 is the property data type $1 is the value of the property $2 is the context_id $3 is the name of the context property
optional delete_context_property = 80
Deletes a property of a context. It has 2 parameters. $0 is the context_id $1 is the name of the context property
optional drop_parent_context_table = 102
Drops the ParentContext table.
optional create_parent_context_table = 103
Creates the ParentContext table.
optional check_parent_context_table = 104
Checks the existence of the ParentContext table.
optional insert_parent_context = 106
Inserts a parent context into the ParentContext table. It has 2 parameters: $0 is the context_id $1 is the parent_context_id
optional select_parent_context_by_context_id = 107
Queries parent contexts from the ParentContext table by context_id. It has 1 parameter. $0 is the context_id
optional select_parent_context_by_parent_context_id = 108
Queries parent contexts from the ParentContext table by parent_context_id. It has 1 parameter. $0 is the parent_context_id
optional select_parent_contexts_by_context_ids = 139
Queries parent contexts from the ParentContext table by context_ids. It has 1 parameter. $0 is the context_ids
optional select_parent_contexts_by_parent_context_ids = 140
Queries parent contexts from the ParentContext table by parent_context_ids. It has 1 parameter. $0 is the parent_context_ids
optional drop_event_table = 35
Drops the Event table.
optional create_event_table = 36
Creates the Event table.
optional check_event_table = 50
Checks the existence of the Event table.
optional insert_event = 37
Inserts an event into the Event table. It has 4 parameters. $0 is the artifact_id $1 is the execution_id $2 is the event type $3 is the event time
optional select_event_by_artifact_ids = 96
Queries events from the Event table by a collection of artifact ids. It has 1 parameter. $0 is the collection string of artifact ids joined by ", ".
optional select_event_by_execution_ids = 97
Queries events from the Event table by a collection of execution ids. It has 1 parameter. $0 is the collection string of execution ids joined by ", ".
optional drop_event_path_table = 40
Drops the EventPath table.
optional create_event_path_table = 41
Creates the EventPath table.
optional check_event_path_table = 51
Checks the existence of the EventPath table.
optional insert_event_path = 42
Inserts a path into the EventPath table. It has 4 parameters $0 is the event_id $1 is the step value case, either index or key $2 is the is_index_step indicates the step value case $3 is the value of the step
optional select_event_path_by_event_ids = 98
Queries paths from the EventPath table by a collection of event ids. It has 1 parameter. $0 is the collection string of event ids joined by ", ".
optional drop_association_table = 81
Drops the Association table.
optional create_association_table = 82
Creates the Association table.
optional check_association_table = 83
Checks the existence of the Association table.
optional insert_association = 84
Inserts an association into the Association table. It has 2 parameters. $0 is the context_id $1 is the execution_id
optional select_association_by_context_id = 85
Queries association from the Association table by its context id. It has 1 parameter. $0 is the context_id
optional select_associations_by_execution_ids = 142
Queries associations from the Association table by execution ids. It has 1 parameter. $0 are the execution_ids.
optional drop_attribution_table = 87
Drops the Attribution table.
optional create_attribution_table = 88
Creates the Attribution table.
optional check_attribution_table = 89
Checks the existence of the Attribution table.
optional insert_attribution = 90
Inserts an attribution into the Attribution table. It has 2 parameters. $0 is the context_id $1 is the artifact_id
optional select_attribution_by_context_id = 91
Queries attribution from the Attribution table by its context id. It has 1 parameter. $0 is the context_id
optional select_attributions_by_artifact_ids = 143
Queries attributions from the Attribution table by artifact ids. It has 1 parameter. $0 are the artifact_ids.
optional drop_mlmd_env_table = 60
Drops the MLMDEnv table.
optional create_mlmd_env_table = 61
Creates the MLMDEnv table.
int64 schema_version = 59
The version of the current query config. Increase the version by 1 in any CL that includes physical schema changes and provides a migration function that uses a list migration queries. The database stores it to indicate the current database version. When metadata source creates, it compares the given `schema_version` in query config with the `schema_version` stored in the database, and migrate the database if needed.
optional check_mlmd_env_table = 63
Checks the MLMDEnv table and query the schema version. At MLMD release v0.13.2, by default it is v0.
optional check_mlmd_env_table_existence = 141
This query returns an int 0 or int 1 to indicate whether mlmdenv table and the corresponding schema exist. In contrast, check_mlmd_env_table will return the schema_version value directly, assuming mlmdenv table exists. The reason for this query is that PostgreSQL will abort transaction if a query inside that transaction fails, while other DB types do not abort the transaction. This query serves single purpose to check whether mlmdenv table exists, while check_mlmd_env_table serves two goals: table schema checking and query for MLMD schema version.
optional insert_schema_version = 66
Insert schema_version. $0 is the schema_version
optional update_schema_version = 64
Update schema_version $0 is the schema_version
optional check_tables_in_v0_13_2 = 65
Check the database is a valid database produced by 0.13.2 MLMD release. The schema version and migration are introduced after that release.
repeated secondary_indices = 105
A list of secondary indices to be applied on the current schema. This is intended for indices that cover multiple columns or which cannot be created as part of table DDL statements.
map<int64, > migration_schemes = 62
Each metadata source should provides migration schemes, each of which defines the schema change details for a particular `schema_version` (sv_i). When a migration procedure wants to upgrade to sv_i, it looks for the MigrationScheme with sv_i as map key.
optional delete_contexts_by_id = 112
Deletes contexts by id. $0 are the context ids.
optional delete_contexts_properties_by_contexts_id = 113
Deletes contexts properties by contexts ids. $0 are the context ids.
optional delete_parent_contexts_by_parent_ids = 114
Delete parent contexts by parent context ids. $0 are the parent context ids.
optional delete_parent_contexts_by_child_ids = 115
Delete parent contexts by child context ids. $0 are the child context ids.
optional delete_parent_contexts_by_parent_id_and_child_ids = 131
Delete parent contexts by parent context ids. $0 is the parent context id. $1 are the child context ids.
optional delete_artifacts_by_id = 116
Deletes artifacts by id. $0 are the artifact ids.
optional delete_artifacts_properties_by_artifacts_id = 117
Deletes artifacts properties by artifacts ids. $0 are the artifact ids.
optional delete_executions_by_id = 118
Deletes executions by id. $0 are the execution ids.
optional delete_executions_properties_by_executions_id = 119
Deletes executions properties by executions ids. $0 are the execution ids.
optional delete_events_by_artifacts_id = 120
Delete events by artifact ids. $0 are the artifact ids.
optional delete_events_by_executions_id = 121
Delete events by execution ids. $0 are the execution ids.
optional delete_associations_by_contexts_id = 122
Delete associations by context ids. $0 are the context ids.
optional delete_associations_by_executions_id = 123
Delete associations by execution ids. $0 are the context ids.
optional delete_attributions_by_contexts_id = 124
Delete attributions by context ids. $0 are the context ids.
optional delete_attributions_by_artifacts_id = 125
Delete attributions by artifact ids. $0 are the artifact ids.
optional delete_event_paths = 126
Delete event paths. This query cleans up event paths where the events do not exist.
optional delete_parent_type = 127
Delete a parent type from the ParentType table. It has 2 parameters: $0 is the type_id $1 is the parent_type_id
optional select_properties_by_type_id = 129
Queries properties of a list of types from the TypeProperty table by `type_ids`. Returns a list of properties(type_id, name, data_type).It has 1 parameter: $0 is the type_ids
optional metadata_source_type_specific_options = 135
Allows different MetadataSourceTypes to configure its own options when creating MetadataAccessObject.
message
metadata_source.proto:617Used in:
int32 total_num_tables = 1
Total number of MLMD tables.
int32 total_num_indexes = 2
Total number of MLMD indexes.
message
metadata_source.proto:628A migration scheme that is used by a migration function to transit a database at a schema_version to schema_version + 1. DDL is often metadata source specific, if provided, each metadata source should have its own setting.
Used in:
repeated upgrade_queries = 1
Sequence of queries to increase the schema version by 1.
repeated downgrade_queries = 3
Sequence of queries to decrease the schema version by 1.
optional upgrade_verification = 2
Details of verifying the correctness of upgrade_queries.
optional downgrade_verification = 4
Details of verifying the correctness of downgrade_queries.
optional db_verification = 5
Verification of the DB schema in this version.
message
metadata_source.proto:637For test purposes, it defines the setup query and post condition invariants of a migration scheme.
Used in:
repeated previous_version_setup_queries = 1
This optional field defines the additional details of recreating: a) the schema of previous version, as the DDL queries will not be available in newer library versions. b) and/or the records of related tables which have schema changes. Note the setup queries is used in a sequence along with other migration schemes of previous version in order to test upgrading from all known previous versions to the library version.
repeated post_migration_verification_queries = 2
This optional field defines the verification queries, each of which returns only True/False in the select to assert the state transition invariant is the same, e.g., * conditions on number of rows * conditions on data model entities (e.g., type, artifact, property).
message
metadata_source.proto:65Template of a SQL query, which can contain parameterized variables using $0, $1, ... $9. For instance: query: "select * from foo where bar = $0" parameter_num: 1
Used in: , ,
string query = 1
int32 parameter_num = 2
Contains supported metadata sources types in MetadataAccessObject. Next index: 7
Used in:
UNKNOWN_METADATA_SOURCE = 0
FAKE_METADATA_SOURCE = 1
a fake in memory metadata_source for testing DEPRECATED -- use SQLITE_METADATA_SOURCE instead.
MYSQL_METADATA_SOURCE = 2
a MYSQL metadata source.
SQLITE_METADATA_SOURCE = 3
A Sqlite metadata source.
POSTGRESQL_METADATA_SOURCE = 6
PostgreSQL, the index number is related to ConnectionConfig.
message
metadata_store.proto:820Configuration for the gRPC metadata store client.
optional string host = 1
The hostname or IP address of the gRPC server. Must be specified.
optional int64 port = 2
The TCP Port number that the gRPC server accepts connections on. Must be specified.
optional ssl_config = 3
Configuration for a secure gRPC channel. If not given, insecure connection is used.
optional channel_arguments = 4
GRPC channel creation arguments.
optional double client_timeout_sec = 5
Time duration that a client is willing to wait for a reply from the server. If unset, the timeout is considered infinite. When the field is specified, Grpc APIs would return DeadlineExceededError when server does not respond within `client_timeout_sec`. Floating point valued, in seconds.
message
metadata_store.proto:827Used in:
optional string client_key = 1
The PEM-encoded private key as a byte string, or Empty if no private key should be used.
optional string server_cert = 2
The PEM-encoded certificate chain as a byte string to use or or Empty if no certificate chain should be used.
optional string custom_ca = 3
The PEM-encoded root certificates as a byte string, or Empty to retrieve them from a default location chosen by gRPC runtime.
message
metadata_store.proto:855Configuration for the gRPC metadata store server.
optional connection_config = 1
Configuration to connect the metadata source backend.
optional migration_options = 3
Configuration for upgrade and downgrade migrations the metadata source.
optional ssl_config = 2
Configuration for a secure gRPC channel. If not given, insecure connection is used.
message
metadata_store.proto:862Used in:
optional string server_key = 1
Private server key for SSL
optional string server_cert = 2
Public server certificate
optional string custom_ca = 3
Custom certificate authority
optional bool client_verify = 4
Valid client certificate required?
message
metadata_store.proto:759Used in:
optional bool enable_upgrade_migration = 3
If not set, by default the upgrade migration is disabled. MLMD only compares db_v with the lib_v, and raise error if the two do not align. If the field is set to true, MLMD performs upgrade migration. It upgrades the database schema version (db_v) to align with the library schema version (lib_v) when connecting to the database. Schema migration should not be run concurrently with multiple clients to prevent data races.
optional int64 downgrade_to_schema_version = 2
Downgrade the given database to the specified schema version. For v0.13.2 release, the schema_version is 0. For 0.14.0 and 0.15.0 release, the schema_version is 4. More details are described in g3doc/get_start.md#upgrade-mlmd-library Set this field only when a database is accidentally upgraded by a newer version library. Each library version only knows how to downgrade to previous schema versions. As downgrade migrations inevitably introduce data loss, please consider taking a backup of the database before downgrading schema. After downgrade migration, the database connection is canceled. The user needs to downgrade the library to use the database.
message
metadata_store.proto:607Used in:
optional string host = 1
The hostname or IP address of the MYSQL server: * If unspecified, a connection to the local host is assumed. The client connects using a Unix socket specified by `socket`. * Otherwise, TCP/IP is used. Currently a replicated MYSQL backend is not supported.
optional int64 port = 2
The TCP Port number that the MYSQL server accepts connections on. If unspecified, the default MYSQL port (3306) is used.
optional string database = 3
The database to connect to. Must be specified. After connecting to the MYSQL server, this database is created if not already present unless skip_db_creation is set. All queries after Connect() are assumed to be for this database.
optional string user = 4
The MYSQL login id. If empty, the current user is assumed.
optional string password = 5
The password to use for `user`. If empty, only MYSQL user ids that don't have a password set are allowed to connect.
optional string socket = 6
The Unix socket to use to connect to the server. If unspecified, a `host` must be provided.
optional ssl_options = 7
If the field is set, the ssl options are set in mysql_options before establishing a connection. It is ignored if the mysql server does not enable SSL.
optional bool skip_db_creation = 8
A config to skip the database creation if not exist when connecting the db instance. It is useful when the db creation is handled by an admin process, while the lib user should not issue db creation clauses.
message
metadata_store.proto:633The options to establish encrypted connections to MySQL using SSL.
Used in:
optional string key = 1
The path name of the client private key file.
optional string cert = 2
The path name of the client public key certificate file.
optional string ca = 3
The path name of the CA certificate file.
optional string capath = 4
The path name of the directory that contains trusted SSL CA certificates.
optional string cipher = 5
The list of permissible ciphers for SSL encryption.
optional bool verify_server_cert = 6
If set, enable verification of the server certificate against the host name used when connecting to the server.
message
metadata_source.proto:756A payload that can be optionally attached to absl::Status messages to indicate failure specific information like error codes for MySQL based backends.
int64 mysql_error_code = 1
Error code for the error.
message
metadata_store.proto:574The only member of this type is a None artifact. Note: ArtifactStruct{} is a None artifact. This can represent an execution that has no outputs (or inputs), or can be part of a UnionArtifactStructType to represent an optional input. For example, StatsGen has an "optional" schema input. A practical example of this is: stats_gen_type = { "dict":{ "properties":{ "schema":{ "union_type":{ "none":{}, "simple":{...schema type...} }, }, "data":{ "simple":{...data_type...} } } } };
Used in:
(message has no fields)
message
metadata_store.proto:465the Parental Context edges between Context and Context instances.
Used in: ,
optional int64 child_id = 1
optional int64 parent_id = 2
optional system_metadata = 3
Output only.
message
metadata_store.proto:694A config contains the parameters when using with PostgreSQLMetadatSource. Next index: 10
Used in:
optional string host = 1
Name of host to connect to. If the host name starts with /, it is taken as a Unix-domain socket in the abstract namespace.
optional string hostaddr = 2
Numeric IP address of host to connect to. If this field is provided, `host` field is ignored.
optional string port = 3
Port number to connect to at the server host, or socket file name extension for Unix-domain connections.
optional string user = 4
PostgreSQL user name to connect as. Defaults to be the same as the operating system name of the user running the application.
optional string password = 5
Password to be used if the server demands password authentication.
optional string passfile = 6
Specifies the name of the file used to store passwords.
optional string dbname = 7
The database name. Defaults to be the same as the user name.
optional bool skip_db_creation = 8
A config to skip the database creation if not exist when connecting the db instance. It is useful when the db creation is handled by an admin process, while the lib user should not issue db creation clauses.
optional ssloption = 9
message
metadata_store.proto:726Used in:
optional string sslmode = 1
disable, allow, verify-ca, verify-full, etc. Reference: https://www.postgresql.org/docs/current/libpq-connect.html
optional string sslcert = 2
This parameter specifies the file name of the client SSL certificate, replacing the default ~/.postgresql/postgresql.crt. This parameter is ignored if an SSL connection is not made.
optional string sslkey = 3
This parameter specifies the location for the secret key used for the client certificate. It can either specify a file name that will be used instead of the default ~/.postgresql/postgresql.key, this parameter is ignored if an SSL connection is not made.
optional string sslpassword = 4
This parameter specifies the password for the secret key specified in sslkey, allowing client certificate private keys to be stored in encrypted form on disk even when interactive passphrase input is not practical.
optional string sslrootcert = 5
This parameter specifies the name of a file containing SSL certificate authority (CA) certificate(s). If the file exists, the server's certificate will be verified to be signed by one of these authorities. The default is ~/.postgresql/root.crt.
The list of supported property value types.
Used in: , ,
UNKNOWN = 0
INT = 1
DOUBLE = 2
STRING = 3
STRUCT = 4
Prefer to use `PROTO` to store structed data since this option has inefficient database storage usage.
PROTO = 5
BOOLEAN = 6
message
metadata_store_service.proto:56Used in:
optional bool abort_if_latest_updated_time_changed = 1
When there are multiple writers to update an existing node to different states, there may be a race and the end result of the concurrent update is nondeterministic. If the field is set, then an optimistic concurrency control (OCC) scheme is used during update: it compares the `artifact`.`last_update_time_since_epoch` in the request with the stored `last_update_time_since_epoch` having the same `artifact`.`id`. If they are different, the request fails, and the user can read the stored node and retry node update. When the option is set, the timestamp after update is guaranteed to be increased and different from the input artifact. When set the option, the caller should set it for all concurrent writers.
message
metadata_store_service.proto:292A pair of an artifact and an event used or generated by an execution, e.g., during the execution run, it uses none or many artifacts as input, and generate none or many artifacts as output.
Used in:
optional artifact = 1
The pair may have an artifact. If present and no artifact.id is given, then it inserts the artifact, otherwise it updates the artifact.
optional event = 2
The pair may have an event. Providing event.artifact_id or event.execution_id is optional. If the ids are given, it must align with the `artifact`.id / `execution`.id respectively. If artifact is not given and event.artifact_id is set, it must exist in the backend.
message
metadata_store_service.proto:303Used in:
optional bool reuse_context_if_already_exist = 1
When there's a race to publish executions with a new context with the same context.name, by default there'll be one writer succeeds and the rest of the writers returning AlreadyExists errors. If set to true, the API will reuse the stored context in the transaction and perform an update.
optional bool reuse_artifact_if_already_exist_by_external_id = 2
When there's a race to publish executions with a new artifact with the same artifact.external_id, by default there'll be one writer succeeds and the rest of the writers returning AlreadyExists errors. If set to true and an Artifact has non-empty external_id, the API will reuse the stored artifact in the transaction and perform an update. Otherwise, it will fall back to relying on `id` field to decide if it's update (if `id` exists) or insert (if `id` is empty).
optional bool force_update_time = 3
When `force_update_time` is set to true, `execution.last_update_time_since_epoch` is force-updated even if input execution is the same as stored execution.
optional bool force_reuse_context = 4
If true, for contexts with a context.id, the stored context will NOT be updated. For such contexts, we will only look at the context.id. We will validate that it exists, and associate the context with the execution. If the context does not exist, a NotFound error is returned.
message
metadata_store_service.proto:366Used in:
optional int32 execution_index = 1
Index in the array of executions.
optional int32 artifact_index = 2
Index in the array of artifacts.
optional event = 3
message
metadata_store_service.proto:375Used in:
optional bool reuse_context_if_already_exist = 1
When there's a race to publish executions with a new context with the same context.name, by default there'll be one writer succeeds and the rest of the writers returning AlreadyExists errors. If set to true, the API will reuse the stored context in the transaction and perform an update.
optional bool reuse_artifact_if_already_exist_by_external_id = 2
When there's a race to publish executions with a new artifact with the same artifact.external_id, by default there'll be one writer succeeds and the rest of the writers returning AlreadyExists errors. If set to true and an Artifact has non-empty external_id, the API will reuse the stored artifact in the transaction and perform an update. Otherwise, it will fall back to relying on `id` field to decide if it's update (if `id` exists) or insert (if `id` is empty).
message
metadata_source.proto:23A collection of returned records.
repeated string column_names = 1
index-aligned column names for all records
repeated records = 2
a list of records returned by a query
message
metadata_source.proto:26An individual record (e.g., row) returned by a MetadataSource. The record does not address the type conversion.
Used in:
repeated string values = 1
message
metadata_store.proto:785Used in:
optional int64 max_num_retries = 1
The max number of retries when transaction returns Aborted error.
message
simple_types.proto:25A bundle of ml-metadata types to describe artifacts, executions and contexts in general ML pipelines. The details of the data model is described in go/mlmd. ml-metadata provides a predefined bundle defined in simple_types_constants.h.
repeated artifact_types = 1
A list of artifact types.
repeated execution_types = 2
A list of execution types.
repeated context_types = 3
A list of context types.
message
metadata_store.proto:660A config contains the parameters when using with SqliteMetadatSource.
Used in:
optional string filename_uri = 1
A uri specifying Sqlite3 database filename, for example: file:some_sqlite3_file_in_local_dir.db file:///home/username/some_sqlite3_file.db see https://www.sqlite.org/c3ref/open.html for model details If not given, a in-memory sqlite3 database is used, and destroyed when disconnecting the metadata source.
optional SqliteMetadataSourceConfig.ConnectionMode connection_mode = 2
A flag specifying the connection mode. If not given, default connection mode is set to READWRITE_OPENCREATE.
Connection parameters for SQLite3 based metadata source.
Used in:
UNKNOWN = 0
READONLY = 1
Connect a metadata source in read-only mode. Connection fail if the sqlite3 database at the `filename` does not exist. Any queries modifying the database fail.
READWRITE = 2
Connect a metadata source in read/write mode. Connection fail if the sqlite3 database at the `filename` does not exist.
READWRITE_OPENCREATE = 3
Similar to READWRITE. In addition, it creates the database if it does not exist.
message
metadata_store.proto:28optional string type_name = 1
The name of a system defined type.
message
metadata_store.proto:1064Options for transactions. Note: This is under development. Clients should not use it.
Used in: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
optional string tag = 1
Transaction tag for debug use only.
message
metadata_store.proto:582An ordered list of heterogeneous artifact structs. The length of the list is fixed. Each position in the list can have a different type.
Used in:
repeated elements = 1
message
metadata_store.proto:511Represents a union of types.
Used in:
repeated candidates = 1
An artifact struct matches this type if it matches any of the candidates. If candidates is empty, this is a bottom type (matches no artifacts).
message
metadata_store.proto:34A value in properties.
Used in: , ,
oneof value
int64 int_value = 1
double double_value = 2
string string_value = 3
struct_value = 4
proto_value = 5
bool bool_value = 6