Proto commits in instill-ai/pipeline-backend

These 44 commits are when the Protocol Buffers files have changed:

2026-01-23

refactor(api): adopt flattened namespace prefixes from protobufs (#1153) **Because** - The protobufs repository has been updated to remove redundant "Namespace" prefixes from Pipeline and Model APIs - The global `ListPipelines` has been renamed to `ListPublicPipelines` to distinguish from namespace-scoped listing - Model trigger API naming has been aligned with Pipeline convention (`TriggerModel` for latest, `TriggerModelVersion` for specific) - Artifact API field names have been updated to follow AIP-122 (`knowledge_base_id` → `knowledge_base`) **This commit** - Updates all Pipeline handler, service, and repository code to use new API names: - `ListNamespacePipelines` → `ListPipelines` - `CreateNamespacePipeline` → `CreatePipeline` - `GetNamespacePipeline` → `GetPipeline` - `UpdateNamespacePipeline` → `UpdatePipeline` - `DeleteNamespacePipeline` → `DeletePipeline` - `ValidateNamespacePipeline` → `ValidatePipeline` - `RenameNamespacePipeline` → `RenamePipeline` - `CloneNamespacePipeline` → `ClonePipeline` - `TriggerNamespacePipeline` → `TriggerPipeline` - `TriggerAsyncNamespacePipeline` → `TriggerAsyncPipeline` - And all related Pipeline Release APIs - Renames global listing endpoint: - `ListPipelines` (global) → `ListPublicPipelines` in handler, service, and repository layers - Updates instillmodel component to use new Model API names: - `TriggerNamespaceModel` → `TriggerModel` - `TriggerNamespaceModelRequest` → `TriggerModelRequest` - `TriggerNamespaceModelResponse` → `TriggerModelResponse` - Updates instillartifact component to use new field names: - `KnowledgeBaseId` → `KnowledgeBase` - Updates integration tests to use new RPC method names - Regenerates mock files to reflect interface changes - Updates protogen-go dependency via local replace directive

The documentation is generated from this commit.

2026-01-21

refactor(api): update tests for AIP-compliant APIs and add private service tests (#1151) **Because** - Integration tests were using deprecated `*UserPipeline` APIs that have been removed in favor of `*NamespacePipeline` APIs - The `LookUpPipeline` endpoint was incorrectly exposed in public service despite being marked as INTERNAL - Private service APIs (LookUpPipelineAdmin, ListPipelinesAdmin) lacked integration test coverage - Several tests were skipped due to outdated API method references - The `PipelineRun.name` field was returning empty namespace/pipeline values **This commit** - Updates all gRPC and REST integration tests to use new Namespace APIs (`CreateNamespacePipeline`, `TriggerNamespacePipeline`, etc.) - Removes public `LookUpPipeline` handler since it was removed from the public protobuf service - Adds private service integration tests (`grpc-pipeline-private.js`) with proper service-to-service authentication using `Instill-User-Uid` and `Instill-Requester-Uid` headers - Fixes `PipelineRun` to properly populate the `Pipeline` field before conversion, ensuring correct resource name generation - Updates `renamePipelineRequiredFields` to include `new_pipeline_id` per AIP-136 - Adds database query helpers (`getNamespaceUidFromId`, `getPipelineUidFromId`) for internal UID lookups - Renames JWT auth test files to Basic Auth to reflect CE edition authentication - Updates integration test protobufs to match current API definitions

2026-01-18

refactor(pipeline): implement AIP-compliant resource naming with hash-based IDs (#1149) **Because** - The codebase previously used user-provided IDs for resources, which could lead to naming conflicts and didn't follow AIP (API Improvement Proposals) standards - The `*User*` and `*Organization*` scoped handlers were deprecated in favor of unified `*Namespace*` scoped handlers - The usage reporting module is now handled at a different layer (EE) - Integration test proto structure needed to align with the flattened protobuf package structure **This commit** - **Database migrations (000043-000050)**: Adds `slug` columns to pipeline, pipeline_release, secret, and connection tables, then migrates from user-provided IDs to server-generated hash-based canonical IDs (e.g., `pip-xxxx`, `sec-xxxx`, `con-xxxx`) - **Handler cleanup**: Removes deprecated `*User*` and `*Organization*` scoped handlers (~525 lines removed): - `CheckName`, `RenameUserPipelineRelease`, `RenameOrganizationPipelineRelease` - `RestoreUserPipelineRelease`, `RestoreOrganizationPipelineRelease` - Related helper functions and interfaces - **Removes usage reporting module** (`pkg/usage/usage.go`): 270 lines removed, functionality moved to EE layer - **Integration test restructuring**: - Flattens proto directory structure (`pipeline/pipeline/v1beta/` → `pipeline/v1beta/`) - Consolidates shared proto files (`google/`, `mgmt/`, `protoc-gen-openapiv2/`) - Removes obsolete test files for deprecated private endpoints - **Component updates**: Updates artifact and model component clients to use new protobuf package paths

2025-06-16

chore(proto): adopt latest `protogen-go` package (#1024) Because - we've updated the enum for user plan. This commit - adopts latest `protogen-go` package. - removed unused mock files.

2025-03-27

chore(all): update domain name in docs (#992) This commit - updates domain name in docs

chore(all): update domain name in docs (#992) This commit - updates domain name in docs

2025-02-04

feat(all): rename VDP to pipeline (#963) This commit - renames `vdp` to `pipeline`

2024-11-29

feat(connection): add lookup connection endpoint (#888) Because - Agent configurations will store the UIDs of the connections needed to use in agent tools (UIDs are used to avoid breaking the config if the ID is updated by the namespace). This commit - Adds a private endpoint to fetch a connection by UID. Part of INS-6933, branch name is wrong.

chore(mod): update to the latest protogen-go package and proto files (#887) This commit - updates to the latest protogen-go package and proto files

feat(trigger): accept connection references in the pipeline trigger data (#883) Because - When a pipeline recipe references a connection, that connection ID will be fetched from the requester's namespace. - This forces requesters to create connections in order to trigger pipelines from other namesapces. This commit - Adds a `connectionReferences` field in the pipeline trigger data, which allows clients to override the references in the recipe with per-batch references. - Adds a private endpoint to fetch connections by UID, needed in the 1-minute to impact project.

2024-11-27

chore(integration): remove deprecated `schemas` field (#877) Because - The `schemas` field was added under the wrong assumption that the integration schemas would differ across connection methods (OAuth vs key-value). - After deprecating this field and providing an alternative `setupSchema` field, all the clients for the deprecated one have been removed. Therefore, removing the deprecated field won't break any client. This commit - Removes the deprecated `schemas` field.

2024-10-30

chore(integration): read namespace ID from creation / update request, not object (#783) Because - Namespace ID in connection requests was being read from the nested object, which caused conflicts in the OpenAPI definitions. This commit - Reads the namespace ID as an explicit parameter. This field in the Connection object becomes, then, an `OUTPUT_ONLY` field.

2024-10-04

feat(integration): add identity to OAuth connections (#699) Because - When a user authorizes an Instill component using OAuth, we want to display the identity that they used. This commit - Adds identity information to OAuth connections. - Makes OAuth method the only accepted one for connections if it's supported.

2024-09-22

feat(integration): support OAuth connections (#661) Because - Some components support OAuth 2.0 integrations to fill their setup. This commit - Allows `METHOD_OAUTH` in connections and OAuth-specific fields. - Returns single setup for integrations.

2024-09-06

feat(integration): reference connections in pipelines (#638) Because - After creating a connection, users need to be able to reference it from a pipeline. This commit - (fix) Makes connection ID mutable - Resolves connection references on pipelines (e.g. `setup: ${connection.my-cohere-conn}`).

2024-09-02

feat(integration): add create, update, delete connection endpoints (#629) Because - Users need to create connections before referencing them in pipelines. Also, they need to be able to modify and delete them. This commit - Add connection creation endpoint.

2024-08-30

feat: add endpoints to read integrations (#611) Because - Users need to know the available integrations in order to create or modify one. This commit - Adds the integration Read enptints (get, list). Information is extracted from the component definition whose ID matches the integration ID.

2024-07-25

feat: use explicit `user_id` and `organization_id` in mgmt request (#559) Because - Previously, we used a general `name` parameter in request parameters, e.g., `{name=users/*}`, which was ambiguous. This commit - Uses explicit `user_id` and `organization_id` in request parameters. - Removes unnecessary environment variables.

2024-06-25

feat: allow user to store recipe in YAML format (#524) Because - We are going to support pipeline recipes in YAML format. We will return the recipe in YAML format in the response and also allow users to create/update recipes using YAML format. - Previously, we converted the component definition ID in the recipe into UID when writing it to the database. Now, since we will directly store this raw recipe in the database, we won't do this conversion anymore. In the future, we'll migrate recipes if the component definition ID changes. This commit - Exposes the `raw_recipe` field in the pipeline and pipeline_release endpoints, the original `recipe` in JSON format is still workable. - Removes the component definition ID to UID conversion when reading/writing the recipe from/to the database. Note - Regarding the component definition ID to UID conversion, database migration is needed. We'll open another PR to address this soon. - We only added the most basic integration test for the YAML recipe in this PR. More comprehensive tests will be added after we implement the event trigger pipeline.

2024-05-31

feat: implement new recipe format (#498) Because - We revamped our pipeline recipe format to make it more extensible for different kinds of trigger methods in the future. - We are going to retire the categorization of `connector` and `operator`, unifying all of them into `component`. This commit - Implements the new recipe format. - Refactors the component memory structure. - Adopts the unified component interface. Note - The recipe migration script is not implemented yet; we will add it in the next PR.

2024-04-18

feat: revamp recipe and retire connector resource (#453) Because - We designed a new recipe format to make the recipe self-explanatory by merging the connector resource configuration into the pipeline recipe. We also introduced a new secret management system to allow users to manage their own secrets. This commit - feat: revamp recipe and retire connector - feat: refactor trigger workflow and implement secret override - feat: add component UsageHandler - feat: remove Airbyte connectors - refactor: refactor and clean some unused code

2024-03-11

chore: optimize connector existence check when creating or updating a pipeline (#416) Because - There are many potential bugs in the connector existence check when creating or updating a pipeline. This commit - Optimizes the connector existence check when creating or updating a pipeline.

2024-03-05

feat: revamp recipe format and implement iterator (#405) Because: - The original recipe message utilized `structpb` to encapsulate every configuration, creating an implicit structure that posed a challenge for users or developers to comprehend the data structure. Given the distinct structures associated with various components, it is essential to explicitly expose them to the user. - A special iterator component for array data iteration is desired. This commit: - Revamped the recipe with new Protobuf messages: `StartComponent`, `EndComponent`, `OperatorComponent`, `ConnectorComponent`, and `IteratorComponent`. - Implements iterator. Note: - This is a breaking change in the interface, requiring an update to the implementation on the client-side.

2024-02-27

feat: rename blockchain connector type to application (#397) Because - https://github.com/instill-ai/protobufs/pull/274 removed the Blockchain connector type and modified the connector and operator list view type. - Once https://github.com/instill-ai/connector/pull/131 is merged, `numbers` connector will become an application connector. This commit - Updates the auto-generated proto Go code, adapting the backend to the changes: - Modifies the Blockchain enum value in the database via a migration. - Modifies type conversions to take Application connector into account. - Removes deprecated lookup methods. - Updates `VIEW_XXX` enum values.

2024-02-26

test: use owner profile from mgmtdb as expected result (#395) Because - We will run pipeline integration tests together with other backend tests. The owner profile data might be changed by other backends. We need to use the real data from the database as the expected result instead of using the hardcoded one. This commit - Uses owner profile from mgmtdb as the expected result

2024-02-22

feat: use `mgmtPB.Owner` to embed the owner information in response (#392) Because - Originally, we used `owner_name` and `owner` (`structpb`) to embed owner information for Console. However, `owner` is a `structpb`, and it is not well-formatted. Since we already have dedicated messages for `User` and `Organization`, we should utilize these messages. This commit - Utilizes `mgmtPB.Owner` to embed the owner information in pipeline and connector response when `view != VIEW_BASIC`. - Updates Connector and Operator packages.

2024-01-05

feat: remove controller-vdp (#354) Because - We used controller-vdp to monitor the connector and pipeline real state, but it required extra computational resources and did not yield benefits. Therefore, we have decided to retire the controller-vdp. This commit - Remove controller-vdp related code

2023-12-06

chore(deps): upgrade protobufs and endpoint version to beta (#312) Because - we are going to release beta version of VDP This commit - upgrade protobufs and endpoint version to beta

2023-11-22

refactor(all): merge connector-backend endpoints into pipeline-backend (#296) Because - we're going to retire connector-backend This commit - retire connector-backend endpoints and move these endpoints to pipeline-backend

2023-08-29

chore: refactor datamodel and proto conversion (#220) Because - the boundary of handler and service is not clear. This commit - refactor datamodel and proto conversion

2023-08-25

chore: implement namespace (#219) Because - We would like to make the resources be public or shared to other users. The current structure can not support it since the api endpoint was designed for the users' own resources. This commit - refactor endpoints to support namespace - refactor code structure to have a better boundary among handler, service, repository

2023-08-17

chore: adopt `connector-resources` endpoints (#215) Because - In proto, `Connector` has been renamed to `ConnectorResource` This commit - adopt `connector-resources/` endpoints

2023-08-16

chore: refactor pipeline backend to support component-oriented architecture (#213) Because - we design a new component-oriented architecture to improve vdp This commit - refactor connector-backend for component oriented architecture - support start-operator and end-operator - support form-data input

2023-08-01

chore: adopt new `CONNECTOR_TYPE_DATA` (#209) Because - the connector-backend has merge connector-source/destination into connector-data This commit - adopt new `CONNECTOR_TYPE_DATA` changes

2023-07-19

chore: add test cases for unfinished-recipe pipeline (#204) Because - need test cases for unfinished-recipe pipeline This commit - add test cases for unfinished-recipe pipeline to test activate/deactivate - remove unused code

2023-07-17

chore: rename trigger endpoint and implement GetOperation for TriggerAsync (#200) Because - simplify the naming of `trigger` endpoint - implement a function for user to get the result of async pipeline This commit - rename `trigger` endpoint - implement `GetOperation` endpoint for `TriggerAsync`

2023-07-04

chore: update protobuf and protogen-go

2023-06-28

chore: adopt new pipeline req/resp format (#181) Because - we redefined the pipeline trigger input and output format - the connector endpoints have been refactored - the model-backend has been decoupled from vdp This commit - adopt the new PipelineDataPayload format in TriggerPipeline request/response - adopt the latest connector endpoints - refactor the codebase - support redis as temp storage for both temporal input and output - Note: the triggering function is not working now.

2023-06-01

chore: fix pre-commit warnings (#170) Because - fix pre-commit warnings This commit - fix pre-commit warnings

2023-05-23

chore: separate trigger endpoint to triggerSync and triggerAsync and adopt temporal (#162) Because - need to separate trigger endpoint to triggerSync and triggerAsync - need to guarantee the execution of async job This commit - separate trigger endpoint to triggerSync and triggerAsync - use temporal to handle async trigger job

2023-05-22

chore: adopt new recipe format (#161) Because - need to provide ability to support complex DAG pipeline in the future - need to include pipeline's resource details in response to reduce front-end api call round trips This commit - refactor the recipe format - add recipe format migration - include resource details in recipe to reduce frontend api calls - refactor code structure.

2023-04-19

chore: change owner from using id to uuid (#138) Because - The user ID can be changed now (in Instill Cloud), so using the owner ID is not reliable This commit - refactor owner of a pipeline to use the UUID, not the ID - update protogen-go dependency

2023-04-04

chore: update pipelines recipe with model (#131) Because - we want to remove the concept of model instance This commit - update the recipe with models instead of model_instances fix INS-336

2023-03-20

feat: adopt private/public apis for pipeline (#111) Because - support private APIs used by other services internally This commit - add private APIs - add gRPC test cases Resolves INS-126 Co-authored-by: Ping-Lin Chang <ping-lin.chang@instill.tech>