Proto commits in getdozer/dozer

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2024-03-19

chore: replace metrics with opentelemetry (#2457) * chore: remove live * chore: replace metrics with opentelemetry * chore: change name to monitor context * chore: init application_id * chore: emit connector events * chore: merge commit

The documentation is generated from this commit.

Update and deduplicate a bunch of deps (#2459) * chore: Update and deduplicate a bunch of deps * Remove cloud * Replace dozer_runtime by individual deno crates

2024-02-19

chore: adapt UI for multiple tables in 1 sink (#2408) * chore: adapt ui for multiple tables in 1 sink * Update dozer-cli/src/ui/app/state.rs Signed-off-by: Bei Chu <914745487@qq.com> * fix: fmt --------- Signed-off-by: Bei Chu <914745487@qq.com> Co-authored-by: Bei Chu <914745487@qq.com>

feat: Allow aerospike sink to handle multiple tables (#2393) * feat: Allow aerospike sink to handle multiple tables * fix test

2024-02-10

refactor: remove dozer-api (#2388) * refactor: remove uses of dozer-api * chore: delete dozer-api

2024-02-03

Feat/oracle connector + UI (#2360) * feat: dozer run with ui * chore: DAG ui with Sink * chore: remove start gpts server * chore: remove unused code in app ui * chore: fix compile with latest code * chore: add batch_num to calculate records with batch insert * chore: update metrics * chore: fix rebase code

2024-01-12

fix: Remove `ApiEndpoint::name` to be consistent with other kinds of sinks (#2316) * fix: Remove `ApiEndpoint::name` to be consistent with other kinds of sinks * fix e2e tests * fix compilation * fix test

2023-12-21

feat: Add `CheckpointedLogReader` (#2257)

2023-11-16

feat: Add termination reason and timestamp (#2223)

2023-10-27

refactor: Include `id` in `Record` response (#2156)

2023-10-23

chore: Remove legacy `build_name` and `descriptor_bytes` from `EndpointMeta` (#2172) * chore: Remove proto descriptor from app server `BuildResponse` because it can be generated in API server * chore: Remove unused `InternalPipelineService::ListEndpoints` * chore: Always build to `v0001` and remove build name from cache

2023-10-20

fix: Handle app restart in cache builder (#2162) * fix: Handle app restart in cache builder * chore: Add unit test for `CacheBuildState` * refactor: Address various comments * chore: Seperate reconnect logic and cache creation logic

2023-10-09

chore: include errors if table schemas are unavailable (#2143) * chore: include errors if table schemas are unavailable * implement example for connectors (#2141) * implement example for connectors * address reviews * address reviews --------- Co-authored-by: aaryaattrey <aarya@getdozer.io> * chore: fix conflicts --------- Co-authored-by: Aarya Attrey <74827081+aaryaattrey@users.noreply.github.com> Co-authored-by: aaryaattrey <aarya@getdozer.io>

feat: Use bytes instead of string as file content (#2135) * feat: Use bytes instead of string as file content * Rename variable * Fix clippy * Use proper method

2023-10-08

chore: fix ls-files and cli experience (#2133) * chore: refactor lockfile path * chore: refactor cloud config * chore: refactor config initiation * chore: store and clean up * chore: include get_all_schemas * chore: add default

2023-10-07

feat: refactor list_tables (#2130)

2023-09-28

Add delete version api (#2102)

refactor: Use `version` instead of `deployment` in all APIs (#2091)

2023-09-27

chore: update proto for infastructure info (#2092)

2023-09-26

fix: rm_alias return type (#2088)

Add version aliasing (#2087)

2023-09-25

Remove UpsertVersion, as it does not make sense anymore (#2084)

chore: include simple connection info in AppResponse (#2075) * add connection info in AppResponse * chore: update to lowercase * chore: remove duplicate code

feat: Add get configuration API (#2081)

2023-09-22

feat: Implement separate function for API request samples (#2077) * chore: Add API urls * Add sample commands * Minor changes * Implement separate function for API request samples * Remove unnecessary clone * Fix format --------- Signed-off-by: Karolis Gudiškis <karolisgud@gmail.com>

Merge branch 'main' into chore/add-api-urls Signed-off-by: Karolis Gudiškis <karolisgud@gmail.com>

Implement separate function for API request samples

Minor changes

feat: implment connections json schema (#2069) * chore: create json schema from connection types * chore: remove redudant code * chore: fix tests * chore: fix grpc tests

2023-09-21

fix: Add change in #2062 that was pushed after merge queue (#2067) Signed-off-by: Bei Chu <914745487@qq.com>

Add sample commands

chore: Add API urls

2023-09-20

chore/add-read-indicator-for-notification (#2062) * chore/add-read-indicator-for-notification * chore: add application name

2023-09-19

feat: Metrics api (#2013) * Add telemetry api * add range parameters * add query type * get full traces method * Add start and end to traces * make range optional * Remove traces

chore: update proto (#2054)

2023-09-18

chore: refactor cloud methods (#2045) * chore: remove auth for health service * chore: refactor commands * chore: fix clippy * chore: use deployment status instead * chore: fix clippy

2023-09-15

refactor: Remove list files

2023-09-14

feat: Add `subscribe` interface (#2037)

feat: Add instance type to deployment command (#2030) * feat: Add instance type to deployment command * Fix comment * Use infra in start request * Remove CLI params and use infra from configuration * Modify cloud config

Remove CLI params and use infra from configuration

add-time-stamp-in-deployment-info (#2029)

2023-09-13

Use infra in start request

feat: Add instance type to deployment command

chore: remove old methods (#2015) * chore: remove old methods * Change deploy application name * feat: Refactor process printer * Save cloud id * chore: modify deployment status * chore: fix fmt * chore: update proto --------- Co-authored-by: Karolis Gudiškis <karolisgud@gmail.com>

2023-09-12

chore: clean up health check endpoint (#2021) * clean-up-health-check-endpoint * cleanup

2023-09-11

chore: read env DOZER_MASTER_SECRET for api security (#1991) * chore: read env DOZER_MASTER_SECRET for api security * chore: implement GetApiToken for ContractService * chore: move GetApiToken to ApiExplorerService * chore: keep api config in live version * chore: add api_explorer_server * chore: add indicator enable_api_security * fix: clippy * fix: fmt * fix: clippy * chore: update info display * chore: remove duplicate code, prior env over config * fix: missing security override * use as_ref instead of clone * fix: consider DOZER_MASTER_SECRET even in local live state

2023-09-07

change on_event to OnEvent (#2005) * change on_event to OnEvent * edit films.proto * modify .bin file --------- Co-authored-by: aaryaattrey <aarya@getdozer.io>

Add allow-incompatible flag to cloud deploy (#2006)

chore: Remove unused code (#2001) * chore: Remove unused code * chore: Remove unused code

2023-09-01

feat: Implement get_protos (#1963) * feat: impleemnt get_protos * chore: send external libraries * chore: code clean --------- Co-authored-by: chubei <914745487@qq.com>

2023-08-30

feat: Label metrics from live run with "live_run_id" (#1943) * chore: Move progress bar out of `LogReader` * chore: Move `Labels` to `dozer-tracing` * feat: Add global labels to dozer metrics * fix: Pass labels to all metrics * feat: Label metrics from live run with "live_run_id" * fix: Filter progress based on "live_run_id" * chore: style fix

2023-08-29

feat: `on_event` can subscribe to multiple endpoints at once (#1881) * fix: Push event was not enabled if `flags` is not present * feat: `on_event` can subscribe to multiple endpoints at once

2023-08-25

feat: make probabilistic optimizations optional and tunable in the YAML config (#1912) Probabilistic optimization sacrifices accuracy in order to reduce memory consumption. In certain parts of the pipeline, a Bloom Filter is used ([set_processor](https://github.com/getdozer/dozer/blob/2e3ba96c3f4bdf9a691747191ab15617564d8ca2/dozer-sql/src/pipeline/product/set/set_processor.rs#L20)), while in other parts, hash tables that store only the hash of the keys instead of the full keys are used ([aggregation_processor](https://github.com/getdozer/dozer/blob/2e3ba96c3f4bdf9a691747191ab15617564d8ca2/dozer-sql/src/pipeline/aggregation/processor.rs#L59) and [join_processor](https://github.com/getdozer/dozer/blob/2e3ba96c3f4bdf9a691747191ab15617564d8ca2/dozer-sql/src/pipeline/product/join/operator.rs#L57-L58)). This commit makes these optimizations disabled by default and offers user-configurable flags to enable each of these optimizations separately. This is an example of how to turn on probabilistic optimizations for each processor in the Dozer configuration. ``` flags:   enable_probabilistic_optimizations:     in_sets: true # enable probabilistic optimizations in set operations (UNION, EXCEPT, INTERSECT); Default: false     in_joins: true # enable probabilistic optimizations in JOIN operations; Default: false     in_aggregations: true # enable probabilistic optimizations in aggregations (SUM, COUNT, MIN, etc.); Default: false ```

2023-08-24

fix: introduce build states and fix inconsistencies (#1902) * chore: return if state cant be sent * chore: remove broadcasting state when stopped * chore: send build message * chore: fix build

2023-08-23

feat: add endpoint for latest deployment status and quick health check endpoint for application (#1894)

chore: Split ContractService from CodeService (#1890) * chore: Remove `CodeService::GetSql` * chore: Split `ContractService` from `CodeService` * chore: Include deployment number for cloud contract service

2023-08-22

feat: Dozer connector (#1883) * feat: implement dozer connector * Add integration tests for nested dozer connector * Fix most review comments * Add sorting and filtering --------- Co-authored-by: VG <vivekg342@gmail.com>

2023-08-17

feat: implement graph on live ui (#1847) * feat: implement progress * feat: implement enable progress flag * feat: implement progress in live * chore: fix clippy * chore: always use telemetry metrics * fix: Only run build once --------- Co-authored-by: sagar <sagar@getdozer.io> Co-authored-by: chubei <914745487@qq.com>

2023-08-11

feat: Add `list_endpoints` to internal server (#1823)

2023-08-09

chore: add autdit field created_at updated_at (#1836)

2023-08-08

feat: implement live code reload experience. (#1826) * feat: initialize live apis * feat: update live experience * chore: fix errors * chore: include sql * chore: add error when into not provided * chore: comment opening web * chore: introduce into validation * chore: introduce stop sql

feat: synchronize GrpcConfig with proto and add phase in AppResponse (#1830)

2023-08-02

fix: Don't hardcode s3 storage region (#1822)

chore: Add `phase` and `last_error` to `DeploymentStatus` (#1818)

2023-08-01

chore/add slug as critera for fetching account metadata (#1815)

chore: add slug as criteria for company metadata

2023-07-18

chore: Rename more `migrate` to `build` (#1755)

2023-07-13

chore: synchronize schema change (#1744)

2023-07-12

feat: Serve replication log on internal grpc server (#1722) * feat: Use replication log in dozer app * feat: Serve replication log on internal grpc server * feat: Cache builder read log from internal grpc server * fix: Avoid duplicate persisting of log entries * feat: Support replication `LogRequest` timeout * feat: Batch request and preload log in the log reader * fix: Fix log existence check in `needs_migration` * chore: Fix cloud build * chore: Fix test * fix: Log watcher is not triggered on timeout until an op arrives * fix: Add `dozer-log-python` and `dozer-log-js` back * feat: API server reads schema from APP server instead of home dir * fix: Fix `dozer-log-js` and the log related examples

feat: Implement config override (#1736)

2023-07-11

feat: Show deployment logs from deploy API (#1731)

2023-07-10

chore/add list deployments endpoint (#1724)

2023-06-30

fix: Remove secrets value and add secrets support in deploy command (#1697)

2023-06-29

feat: Implement secrets commands (#1695)

2023-06-28

feat: Add cloud secrets API interfaces to cloud.proto (#1689)

2023-06-27

chore: add id in processor factory to support query metric on UI (#1685)

2023-06-22

Revert "chore: enhance response for UI render (#1648)" (#1664) This reverts commit 94136c1d03cac7eb14f7376f81b63fc87d14a0c4.

2023-06-19

chore: enhance response for UI render (#1648)

2023-05-26

chore: update Company Response metadata (#1603)

2023-05-25

feat: Add deployment number to get status proto file (#1600) * feat: Add deployment number to get status proto file * fix: Find latest deployment number from `DeploymentStatus` --------- Co-authored-by: chubei <914745487@qq.com>

fix: `api_desired` and `api_available` should be optional (#1602)

feat: Add `cloud api set-num-replicas` (#1598)

feat: Add `cloud deploy -n` and `dozer cloud update -n` (#1597)

2023-05-24

chore: Add repository url to cloud proto and rename id to app_id (#1594)

feat: Update `LogMessageRequest` and `LogMessage` to include follow option and the log source (#1590)

2023-05-23

fix: numeric (10, 2) (#1583)

2023-05-22

feat: update cloud proto to create connection and source manually (#1584)

2023-05-19

chore: Rename "revision" to "deployment" (#1579)

chore: add delete files (#1577)

2023-05-18

chore: Remove unused `optional` in `cloud.proto` (#1576)

feat: Support version management (#1575)

feat: Show revision and version information in `cloud status` (#1574)

feature: Add filters and pagination to command (#1573) * feature: Add filters and pagination to command * Fix imports

feature: Add stop app in cloud call (#1572)

2023-05-17

chore/add file management endpoint (#1571)

2023-05-16

feature: Display progress of cloud deployment (#1566)

chore: add grpc endpoint for cloud to support develop ui (#1565)

2023-05-15

chore: update proto model matching dozer types (#1563)

2023-05-12

chore: Remove status update method from internal pipeline server (#1560)

feat: Implement monitoring command (#1559) * feat: Implement monitoring command * Remove println * Fix formatting