Proto commits in ethpandaops/xatu

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2026-06-24

cannon: scaffold proto + migrations + route codegen for 11 new types through Fulu Adds shared/generated artifacts for 11 new cannon event types (Electra execution requests, rewards endpoints, state metadata, Electra state queues): - event_ingester.proto: payload + Additional*Data messages, Event.Name enum values, ClientMeta.additional_data and DecoratedEvent.data oneof fields. - coordinator.proto: CannonType enum values, CannonLocation* messages, CannonLocation.Data oneof fields. - 11 ClickHouse migration pairs (005-015) under deploy/migrations/clickhouse/xatu. - 11 generated canonical_*.gen.go route batches plus compiling route + test scaffolds (empty event names; Phase B wires them in). Also fixes the route generator to read migrations from the xatu/ subdir (broken by the per-schema migration split) and fixes the route scaffold template's package-shadowing bug.

2026-06-22

feat(cannon)!: add execution-layer (EL) dimension via cryo Adds an execution-layer dimension to cannon that extracts canonical execution-layer datasets with the third-party `cryo` binary and writes them straight to ClickHouse, mirroring the per-deriver/per-cursor design of the consensus-layer cannon. Every cryo dataset flows proto DecoratedEvent -> ClickHouse route -> sink, reusing the existing sink layer (no bespoke pipeline). Datasets (16, all cryo-fed): block, transaction, logs, traces, native_transfers, erc20_transfers, erc721_transfers, contracts, balance_diffs, storage_diffs, nonce_diffs, balance_reads, storage_reads, nonce_reads, four_byte_counts, address_appearances. Key pieces: - pkg/cannon/execution/cryo: dataset-agnostic cryo runner (--hex, zstd, column projection) + typed parquet reader. - pkg/cannon/deriver/execution: one deriver per dataset, chunked repeated-payload events, shared internal_index stamping (legacy cumcount parity). - pkg/cannon/iterator/backfilling_block_iterator: CL-finality-gated block iterator (never iterates past the execution block of the CL-finalized beacon block); head walks up, backfill walks down to a floor. State-read datasets are floored to block >=1 (genesis is untraceable by cryo's state tracer). - pkg/clickhouse/route/execution: 16 columnar ClickHouse routes. - proto: 16 ExecutionCanonical* event types + CannonType + CannonLocation wrappers + BackfillingBlockMarker. - Dockerfile: build cryo from a pinned git commit (master exposes gas_limit; pinning gives reproducible builds + stable state-read semantics). Schema fix: canonical_execution_four_byte_counts ORDER BY now includes `signature` -- the prior key collapsed a tx's multiple selectors into one row under ReplacingMergeTree. Validated against the legacy (old-script) prod data across genesis->24M and all fork boundaries: 12/16 datasets byte-identical; the 3 state-reads capture more (newer cryo, desired); four_byte_counts fixed; remaining diffs are prod's known-missing data (the reason for this work) and a single DAO-fork balance_diffs.to_value edge. End-to-end run validated on mainnet (beacon + EL + coordinator + ClickHouse): CL-gating, per-dataset Postgres cursors, and the direct-to-ClickHouse sink all confirmed. Out of scope: transaction structlog (not cryo-fed the same way); extra_data_string for non-UTF8 blocks (proto3 string limit); range-sharded backfill (future). BREAKING CHANGE: the xatu-server output (type: xatu) is now rejected by cannon at config validation, for both CL and EL data -- cannon writes directly to ClickHouse. Existing cannon configs using a `type: xatu` output must switch to a `type: clickhouse` output. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The documentation is generated from this commit.

2026-06-17

fix(libp2p): restore libp2p_join/leave by keying on local_peer_id (#848) * fix(libp2p): use local_peer_id for join/leave routing and dedup libp2p_join and libp2p_leave events come from go-libp2p-pubsub's RawTracer Join(topic)/Leave(topic) callbacks, which carry no remote peer. The clickhouse route validator required a non-empty peer id and dropped every join/leave event (visible as 'nil PeerId: invalid event' in consumoor), so neither table has received data since the clickhouse-raw cutover. Add a local_peer_id field to the join/leave client metadata, populated from the producer's own host ID, and key routing/dedup off it (peer_id_unique_key -> local_peer_id_unique_key). Migration 004 recreates the two tables with the renamed sorting-key column, preserving existing rows via a temporary backup. Pairs with the tysm tracer change that stamps the local host ID onto join/leave. Co-authored-by: original branch feat/libp2p-join-leave-local-peer-id * fix(migration,test): comment-safe migration splitting + goconst constants - Migration 004: remove semicolons from SQL comments. golang-migrate's x-multi-statement splitter is not comment-aware, so a ';' inside the leading comment produced a comment-only chunk and failed with 'code 62, Empty query'. - Tests: extract repeated column/topic literals into shared constants to satisfy goconst. * fix(migration): make 004 database-agnostic Drop hardcoded 'default.' table prefixes and use currentDatabase() in the Distributed engine database arg, matching the db-agnostic migration convention from the per-set migrations refactor. Unqualified table names resolve to the migrator connection's database.

2026-05-29

feat(event_ingester.pb.go): rename event constants and update proto types for consistency with the new API design This change modifies the event constant names and proto types, ensuring they align with the updated API structure. The changes improve clarity and foster easier integration and maintenance of the event ingestion system. refactor(proto): reorder fields in event_ingester.pb.go for improved organization and readability. This change enhances the maintainability of the code by establishing a more logical structure. refactor(proto): rename fields in Event message for clarity and consistency fix(proto): update method signatures in event_ingester_vtproto for renamed fields to match updated Event message structure

Merge branch 'master' into release/gloas * master: feat(sentry-logs): make the xatu sink buffer/concurrency/timeout env-tunable (#846) feat: add state size delta and trie depth (#722) Scope consumoor Kafka traces (#844) lowercase telemetry service names (#843)

2026-05-20

feat: add state size delta and trie depth (#722) Adds execution_state_size_delta and execution_mpt_depth end-to-end: new proto events (#90/#91), consumoor ClickHouse routes, server event-ingester handlers, migration 003_state_metrics, and sentry-logs Vector parsing for geth's statesize tracer output.

2026-05-19

feat(state-metrics): UInt64 counts, schema cleanup, smoke coverage Follow-ups to PR #722. All changes are scoped to the state-metrics work introduced there; the rest of the diff is regenerated artifacts. Proto - ExecutionStateSizeDelta: switched all count/byte wrapper fields from Int64Value to UInt64Value. Writes and deletes are non-negative counts in the upstream geth tracer; signed wrappers were over-permissive. Net delta remains signed in ClickHouse (see below). ClickHouse migration 003_state_metrics - Up: - CREATE TABLE → CREATE TABLE IF NOT EXISTS for idempotent re-runs. - Qualified all table refs with `default.`. - Count/byte columns Int64 → UInt64 to match the proto change. - MATERIALIZED net-delta columns wrap inputs in toInt64(...) before subtraction; UInt64 - UInt64 stays UInt64 in ClickHouse and would wrap on underflow, so the explicit cast is required to preserve negative deltas. - ZooKeeper path normalized to `tables/{shard}/{database}/{table}` to match the convention already used by migrations 001 and 002. - PARTITION BY leads with meta_network_name so per-network operations don't have to scan every block_number bucket. - ORDER BY reordered so meta_network_name precedes block_number; single-network range scans now hit a primary-key prefix lookup. - Down: - Qualified all DROPs with `default.` and added SYNC so the drop blocks until cluster-wide cleanup completes, avoiding a stale replica between migration cycles. Docs - docs/sentry-logs.md: corrected event IDs (88/89 → 90/91) to match the enum values actually allocated in the proto. Smoke test - .github/workflows/sentry-smoke-test.yaml: inject 5 "State metrics" log fixtures alongside the existing block-metrics ones, and add a verification step asserting both execution_state_size_delta and execution_mpt_depth receive rows. Sample queries include the derived account_delta MATERIALIZED column so the toInt64 cast is exercised end-to-end. Regen - pkg/proto/... regenerated against the updated proto. - pkg/clickhouse/route/.../*.gen.go regenerated. The substantive diffs are confined to execution_state_size_delta.gen.go picking up the new UInt64 column types; the remaining file changes are import-block formatting from the codegen toolchain. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-14

Merge branch 'master' into release/gloas Resolves enum + migration collision by shifting BAL+ePBS up by 1: - BEACON_API_ETH_V1_EVENTS_FAST_CONFIRMATION keeps event enum 89, oneof field 95, oneof data field 210, and migration 002. - BEACON_API_ETH_V2_BEACON_BLOCK_ACCESS_LIST moves 89 → 90 (oneof 96/211, migration 003). - All 15 ePBS enums shift up by 1 (90 → 91 ... 104 → 105) with matching oneof field renumbering. Accepts master's consumoor-only local stack (deletes vector-http-kafka.yaml and vector-kafka-clickhouse{,-libp2p}.yaml; adds xatu-consumoor service config + xatu-server direct kafka output with topicPattern "xatu-protobuf-${event-name}"). Bumps ethpandaops/beacon to release/gloas tip 81b9ef3 to pick up both ePBS SSE hooks (release/gloas) and OnFastConfirmation (merged in from master).

refactor(state_size_delta): emit writes/deletes; derive delta in ClickHouse The execution_state_size_delta table previously stored precomputed signed deltas (account_delta, account_bytes_delta, ...). That representation lost information: a block that updates one account looked the same as a block that adds and removes accounts whose totals cancel. This refactor splits each metric into a (writes, deletes) pair carrying the gross churn: - 20 stored Int64 columns covering 5 categories x {count, bytes} x {writes, deletes}: accounts, account trie nodes, contract code, storage slots, storage trie nodes. - 10 MATERIALIZED Int64 columns derived as (writes - deletes) preserve the original delta semantics for any query that wants the net change. An "update" counts as both a write of the new value and a delete of the prev value, so (writes - deletes) recovers the net delta for all three cases (create / update / delete). Contract code remains write-only in geth's state sizer (it is content-addressed and shared across accounts, so deletion would require reference counting). The contract_code_deletes / contract_code_delete_bytes columns are present for schema symmetry and stay 0 until upstream geth grows ref-counting; contract_code_delta therefore equals contract_code_writes. Touches the geth tracer (writes/deletes nested JSON instead of delta), the proto (renamed *_delta fields to *_writes and added *_deletes), the Vector transforms (sentry-logs normaliser + kafka-to-clickhouse VRL), and the ClickHouse migration. Verified end-to-end with the test-geth 1M-block mainnet import: 1,000,001 rows in each table, MATERIALIZED delta identity holds for every row.

populate state tables from logs

feat: capture fast_confirmation beacon-api event in sentry (#830) * feat: capture fast_confirmation beacon-api event end-to-end Wires the new fast_confirmation SSE event (beacon-APIs#598) all the way through xatu: adds the proto schema, the sentry decorator + dedup cache, the server-side ingester registration, a tap on the raw event stream that decorates fast_confirmation messages (since ethpandaops/beacon doesn't yet expose a typed OnFastConfirmation), the ClickHouse table + distributed view as a new 002 migration, the consumoor rowgen and flatten logic, and a cross-cutting TestCompletenessEveryEventNameHasRoute that asserts every Event_Name has a route or an allowlisted reason. The slot field on the event is the confirmed block's slot (typically in the past), so wallclock_slot/wallclock_epoch are also populated from the sentry's wall clock, mirroring voluntary_exit. Triggered by adding fast_confirmation to ethereum.beaconSubscriptions in the sentry config. Depends on ethpandaops/go-eth2-client#24 (currently pinned via pseudo-version; bump to a tag once that merges). * make fast_confirmation derived fields Nullable Epoch 0 is genuine genesis data, not 'missing'. Sentinel-zero on optional derived columns conflated the two. Switch propagation_slot_start_diff, epoch, epoch_start_date_time, wallclock_slot/epoch and their *_start_date_time to Nullable so a real null can mean 'unknown'. slot_start_date_time stays non-null because it's in ORDER BY. * silence goconst on shared column-key test literals * lint: trim unused nolint:goconst markers * bump ethpandaops/beacon to fast_confirmation-aware tip Pulls in ethpandaops/beacon#73 so the library handleEvent switch recognises fast_confirmation, eliminating per-event 'unknown event topic' errors when a sentry subscribes to the topic. * lint: function-level nolint:goconst for snapshot tests * sentry: use beacon.OnFastConfirmation typed handler * bump ethpandaops/beacon to v0.69.0

2026-05-13

feat: add OnPayloadAttestationProcessed observe hook

feat(fast_confirmation): add wallclock_slot and wallclock_epoch The slot field carries the *confirmed* block's slot, which is typically in the past. To know when the confirmation actually arrived at the sentry, mirror the voluntary_exit pattern and add wallclock_slot + wallclock_epoch (with their *_start_date_time siblings) into the ClientMeta additional data, the ClickHouse table, the rowgen, and the sentry decorator.

feat: capture fast_confirmation beacon-api event in sentry Wires the new fast_confirmation SSE event from beacon-APIs#598 through the sentry pipeline. Adds the proto schema (xatu.eth.v1.EventFastConfirmation + BEACON_API_ETH_V1_EVENTS_FAST_CONFIRMATION enum), the sentry decorator and dedup cache, the server-side ingester handler, and an OnEvent tap that filters the raw event stream for fast_confirmation (the beacon library doesn't yet expose a typed OnFastConfirmation publisher). Triggered by adding fast_confirmation to ethereum.beaconSubscriptions in the sentry config; no other config required.

feat: PTC vote three-state breakdown + EIP-7843 slot_number coverage

feat(epbs): wire 2 synthesized observability events from beacon-node internals Adds the producer + consumer side for `BEACON_SYNTHETIC_PAYLOAD_STATUS_RESOLVED` and `BEACON_SYNTHETIC_BUILDER_PENDING_PAYMENT_SETTLEMENT` — TYSM-instrumented events that observe ePBS internal state transitions on every beacon node (multi-witness, not gossip). Naming follows the LIBP2P_TRACE_SYNTHETIC_HEARTBEAT precedent — these are "synthesized" because they don't correspond to any beacon API SSE event; they're emitted from inside prysm via TYSM's hook system. Captured by every TYSM-patched beacon node, every slot / every epoch. What you get from these: - PayloadStatusResolved: empty-slot rate, envelope timing distributions, PTC participation rate, fork-choice latency, INVALID payload transitions. - BuilderPendingPaymentSettlement: builder payment success rate (SETTLED vs DROPPED), lost-revenue totals per builder, quorum-margin distributions, builder market concentration. Changes: - pkg/proto/eth/v1/beacon_synthetic.proto — new PayloadStatusResolved + BuilderPendingPaymentSettlement message types with PayloadStatus (PENDING/FULL/EMPTY/INVALID) and BuilderPendingPaymentOutcome (SETTLED/DROPPED) enums. - pkg/proto/xatu/event_ingester.proto — 2 new Event_Name enum values (102, 103), 2 new AdditionalData messages, 2 new oneof entries in DecoratedEvent (Additional* IDs 108/109, data IDs 223/224). - pkg/clmimicry/ — 2 TraceEvent payload structs, 2 New*Payload builders, 2 TraceEvent_* constants, dispatch map entries, case arms in handleHermesRPCEvent, 2 EventConfig toggles, 2 Group D categorizer registrations, 2 TraceEventTo* converters, 2 handle* methods + 2 deriveAdditionalDataFor* helpers in a new event_beacon_synthetic.go. - pkg/server/service/event-ingester/event/beacon/synthetic/ — 2 new server handler files in a new `synthetic` package (parallels `eth/v1/`); registered in the event router. - deploy/migrations/clickhouse/003_gloas_epbs_support.up.sql — 2 new tables appended (no new migration since 003 is greenfields, not yet deployed). Both use ReplicatedReplacingMergeTree with meta_client_name in the ORDER BY composite to preserve multi-witness records. - pkg/clickhouse/route/beacon/beacon_synthetic_*.go — 2 hand-written FlattenTo routes calling the existing extractBeaconSlotEpochPropagation helper. The matching .gen.go files come from `make clickhouse-routes`. - pkg/clickhouse/route/cmd/generate/manifest.go — 1-line addition registering the `beacon_synthetic_` prefix in the codegen prefix-route table so the generator emits the new files into pkg/clickhouse/route/beacon/. - Updated existing .gen.go files from re-running `make clickhouse-routes` (rowgen tool produces a tighter output than the last commit on a few fields — the regen is consistent with the schema, no functional change). Tests: - pkg/clmimicry/event_categorizer_test.go — counts updated 33→35 events, GroupD ≥10→≥12 (the 2 new synthetic events are Group D — no Topic, no MsgID, same shape as engine API events / custody probes). - pkg/clickhouse/route/beacon/beacon_synthetic_*_test.go — placeholder snapshot tests following the existing `TODO(epbs)` pattern; they currently fail with the standard `"nil … payload: invalid event"` message awaiting real Gloas devnet captures. Verification: - buf lint / buf build → clean - go build ./... → clean - go test -race ./pkg/clmimicry/... → ok - make proto + make clickhouse-routes → clean output Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-08

feat(libp2p): rename peer_id_unique_key to local_peer_id_unique_key on join/leave The shared TraceEventMetadata.peer_id field has overloaded semantics: for most libp2p events it carries the remote peer involved (deliver/graft/prune/gossipsub/etc), but for join/leave it described the local host that subscribed/unsubscribed — there is no remote peer for those callbacks. Reusing the same field name with two meanings made the schema confusing, and pubsub.RawTracer's Join(topic) / Leave(topic) signatures don't even hand producers a peer.ID, so the field was empty in practice and the route validator hard-rejected every event. * Add a dedicated `local_peer_id` string to AdditionalLibP2PTraceJoinData and AdditionalLibP2PTraceLeaveData. * Migration 002 renames the clickhouse column `peer_id_unique_key` to `local_peer_id_unique_key` on libp2p_join_local / libp2p_leave_local (and the Distributed parents). The column is in ORDER BY so RENAME COLUMN is not supported; the migration drops and recreates the tables. * Regenerate route .gen.go files via `make clickhouse-routes`. * Route handlers read only `local_peer_id`; no fallback to the legacy metadata.peer_id. Sentries that haven't been updated will fail validation, by design. * clmimicry.handleJoinEvent / handleLeaveEvent populate the new field from the inbound TraceEvent.PeerID. Pairs with a tysm change that populates TraceEvent.PeerID with the local host's peer.ID for Join/Leave (where the libp2p RawTracer interface provides no peer.ID at all).

feat: add payload_status to ForkChoiceNodeV2

2026-05-07

feat: wire ePBS SSE handlers + cannon derivers

feat(go.proto): add new Gloas beacon block body to handle ePBS and BALs fix(go.mod): update module dependencies including OffchainLabs/go-bitfield for gas price calculations test(eth): implement tests for gas price calculations that support new Gloas structure

migrate to ethpandaops/go-eth2-client; wire ePBS sourcing

2026-03-26

feat: ebps wiring wip

feat(epbs): add server event handlers, libp2p gossipsub types, and ePBS conditional block routing

feat: prepare epbs types

2026-03-25

feat(block_access_list): add support for "touched" event for accounts with no state changes to enhance parallel execution analysis feat(proto): update BlockAccessListChange to include "touched" in change_type options for better clarity on account status

feat(proto): introduce BlockAccessListStorageRead message for enhanced storage read tracking fix(conversion): adjust handling of storage reads to use new BlockAccessListStorageRead structure

feat(gloas): add BAL storage reads and bump deps to bal-devnet-3

2026-03-24

refactor: bals structure rlp decoded

2026-03-22

feat: gloas support

2026-02-23

refactor: trace_identify plumbing

2026-02-20

feat(proto): add LIBP2P_TRACE_IDENTIFY event type to event ingester

feat(proto): add Identify message to libp2p trace proto

refactor: remove deprecated blockprint support entirely (#758) Blockprint is fully deprecated and disconnected from all runtime code paths. Remove all remaining dead code across cannon, server, proto, and config layers to reduce maintenance burden.

2026-02-12

feat: Sentry Logs - Vector-based log collection for execution client metrics (#742) * feat(sentry-logs): add Vector-based client log collection for execution metrics Introduces sentry-logs, a Vector-based log collector that extracts structured metrics from Ethereum execution client logs and sends them to Xatu server. Key features: - Collects geth "Slow block" logs containing execution metrics (timing, cache stats, state I/O) - Multi-stage Vector pipeline: parse JSON → build metadata → route by type → transform → send - HTTP endpoint on xatu-server for receiving events via protojson format - ClickHouse table for execution_block_metrics with full metric breakdown - Docker image published as ethpandaops/xatu:sentry-logs-* Configuration: - Requires XATU_CLIENT_NAME, XATU_NETWORK_NAME, XATU_SERVER_URL, XATU_AUTH env vars - Sources configured via mounted YAML files in sources.d/ - Supports file, docker_logs, and journald source types Also includes: - HTTP ingester service for xatu-server with protojson support - E2E test in sentry smoke test workflow - Documentation at docs/sentry-logs.md * Fix HTTP ingester config fallback and JSON decoding * Ensure sentry-logs sources.d is present * refactor(sentry-logs): make Vector config more composable for multi-client support Restructure the Vector configuration to support multiple execution clients: - Add client detection stage that routes logs based on format patterns - Introduce normalizer pattern: client-specific transforms convert to common schema - Add clear architecture documentation for adding new clients - Handle double-encoded JSON (geth outputs structured events as JSON strings) - Separate concerns: parsing -> detection -> normalization -> event building To add a new client: 1. Add detection condition in `detect_client` route 2. Add normalizer transform (e.g., `normalize_<client>_block_metrics`) 3. Add normalizer to `route_normalized_events` inputs Tested end-to-end with Kurtosis devnet running geth:master with --debug.logslowblock=0, verified all 31 data fields populated in ClickHouse. * fix(sentry-logs): resolve VRL compilation errors for strict Vector validation The VRL compiler was failing with E651 (unnecessary error coalescing) and E103 (unhandled fallible assignment) errors. This was causing the sentry-logs container to fail to start in CI. Changes: - Extract fields into named variables before building the normalized object - Use explicit type coercion (string(), float(), int()) with ?? fallbacks - Handle all optional fields (state_reads, state_writes, cache) with get() * fix(http-ingester): check auth before parsing request body Move authentication check before body reading/parsing to prevent potential DDoS attacks via large payload decompression and parsing from unauthenticated clients. * fix(sentry-logs): update docs and add gzip compression support - Update event ID from 85 to 86 in docs (changed during rebase) - Add XATU_COMPRESSION env var support to Vector config (gzip default) * refactor(http-ingester): reuse services.eventIngester config HTTP ingester now directly uses services.eventIngester configuration instead of having its own nested eventIngester config. This eliminates config duplication and ensures both gRPC and HTTP ingesters share the same auth, outputs, and settings. * docs(server): add HTTP ingester documentation * docs: add httpIngester to example_server.yaml * chore: change httpIngester default port from 8081 to 8087 - Update default port to avoid conflicts with common services - Update all docs, examples, and configs to use new port - Update EXECUTION_BLOCK_METRICS event ID to 87 (bumped during rebase) * fix(filter): silently drop events without modules instead of erroring When an event has no module set and a filter requires specific modules, the filter should silently drop the event rather than returning an error. This allows events like EXECUTION_BLOCK_METRICS (from sentry-logs) to flow through sinks that don't explicitly filter for them, without causing HTTP 500 errors when they hit sinks with modules filters. * refactor(server): extract shared Pipeline from event ingesters Extract the shared event processing infrastructure into a Pipeline struct within the event-ingester package. Both gRPC and HTTP transports now use this Pipeline, eliminating duplicated code for: - Creating authorization - Creating handler and sinks - Starting/stopping sinks - Processing events and sending to sinks This reduces code duplication by ~140 lines and provides a single source of truth for event processing infrastructure.

2026-02-04

feat(cannon): add beacon block sync aggregate deriver Add new cannon type BEACON_API_ETH_V2_BEACON_BLOCK_SYNC_AGGREGATE that extracts sync aggregate data from beacon blocks. Includes: - New deriver for extracting sync committee bits and signature - ClickHouse migration for canonical_beacon_block_sync_aggregate table - Event ingester support for the new event type - Proto definitions for the new event type

fix proto

2026-01-30

feat(cannon): add sync committee deriver Add support for collecting sync committee validator assignments via /eth/v1/beacon/states/{state_id}/sync_committees endpoint. Key changes: - Add BeaconSyncCommitteeDeriver that fetches at period boundaries only (every EPOCHS_PER_SYNC_COMMITTEE_PERIOD epochs) to avoid redundant calls - Add BEACON_API_ETH_V1_BEACON_SYNC_COMMITTEE cannon type and event name - Add SyncCommittee/SyncCommitteeV2 proto messages for validator data - Add ClickHouse migration for canonical_beacon_sync_committee table - Uses Altair as activation fork (when sync committees were introduced)

2026-01-27

Merge master into feat/ethstats-per-node-credential-forwarding - Resolved conflicts in event_ingester.proto - Integrated BEACON_API_ETH_V1_BEACON_BLOB (84) from master - Renumbered ETHSTATS_* event types to 85-91 - Renumbered ethstats data fields to 207-213 (sequential after 206) - Regenerated proto files Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

feat(sentry): add blob sidecar fetching on block receipt (#739) * feat(migrations): add admin.cryo table to migration 049 Add the admin database and cryo table for tracking cryo extraction progress. This matches the platform repo's migration 049. * feat(sentry): add blob sidecar fetching on block receipt When enabled, proactively fetches blob sidecars when blocks are received to ensure blob data is captured even if gossip events are missed. - Add BlobSidecarConfig with enabled flag (default: false) - Add fetchAndEmitBlobSidecars method to fetch and emit blobs - Reuse existing duplicate cache to prevent re-emission - Add converter utility for BlobSidecar to BlobSidecarEvent * docs(example_sentry.yaml): remove redundant comments regarding blob sidecar configuration * refactor(sentry): remove unnecessary goroutine in OnBlock handler for blob fetching * feat(sentry): add BEACON_API_ETH_V1_BEACON_BLOB event type for PeerDAS Add new event type that stores blob metadata derived from block's blob_kzg_commitments. Unlike blob_sidecar which fetches full BlobSidecar with proofs, this stores only metadata including versioned_hash. The versioned_hash enables joining with execution_engine_get_blobs to determine which slot the EL was fetching blobs for - critical for PeerDAS/Fusaka analysis. Changes: - Add Blob message to blob.proto with versioned_hash field - Add BEACON_API_ETH_V1_BEACON_BLOB (84) event type - Add sentry event handler deriving blobs from KZG commitments - Add server event handler for ingestion - Add ClickHouse migration for beacon_blob table - Add Vector pipeline configuration for Kafka routing * refactor(sentry): remove BlobSidecar fetching, extract beacon blob to own file Remove the BlobSidecar fetching functionality that was superseded by the new BeaconBlob event type. Extract fetchAndEmitBeaconBlobs and extractKZGCommitments to their own file for better maintainability. Changes: - Remove BlobSidecarConfig from config.go - Remove fetchAndEmitBlobSidecars from sentry.go - Remove BlobSidecarToBlobSidecarEvent from ethereum/blob.go - Create beacon_blob.go with beacon blob emission logic

Merge master into feat/ethstats-per-node-credential-forwarding Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

2026-01-23

Merge branch 'master' into ralph/horizon

2026-01-22

feat(proto): add returned_blob_indexes to ExecutionEngineGetBlobs (#733) Track which specific blobs were returned (non-null) in engine_getBlobs responses. This is especially useful for V3 which mandates partial responses. Uses Array(UInt8) for storage efficiency (1 byte per index) instead of duplicating hashes (66 bytes each).

2026-01-21

feat: US-018 - Create Horizon module skeleton and CLI command

feat(horizon): US-001 - Add HorizonLocation protobuf message Add HorizonType enum and HorizonLocation message to coordinator.proto for tracking HEAD and FILL slot positions in the Horizon module. Includes RPC methods GetHorizonLocation and UpsertHorizonLocation.

2026-01-20

Merge master into feat/ethstats-per-node-credential-forwarding - Resolved proto conflicts by merging both ExecutionEngine and Ethstats events - Renumbered Ethstats event enum values to avoid conflicts (84-90) - Added RPC_SNOOPER=10, ETHSTATS=11 to module enum - Regenerated proto files Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

2026-01-15

feat: execution engine events (#728) * feat(clickhouse): improve table and column comments across all tables Adds a single migration (089) that systematically improves table and column comments for better documentation and query optimization: - beacon_api_eth_v1_events_* tables: Mention Sentry event-stream subscription - beacon_api non-event tables: Clarify API endpoint calls - canonical_beacon_* tables: Note finalized data and partition keys - libp2p_* tables: Describe deep instrumentation from forked CL clients - mempool_* tables: Note external data imports - mev_relay_* tables: Mention relay data API polling - node_record_* tables: Describe discv5 network crawling Key improvements: - Added partition key info to help users avoid table scans - Changed meta_client_name from "generated" to "collected the data" - Added context about what each row represents - Clarified timing units (milliseconds since slot start) * fix: remove meta_client_name comments from tables without that column Removes COMMENT COLUMN meta_client_name from: - libp2p_peer (lookup table) - mempool_dumpster_transaction (external import) - block_native_mempool_transaction (external import) * feat: Add Execution Engine tables * fix: use UInt32 for duration_ms and improve ORDER BY in execution engine tables - Change duration_ms from UInt64 to UInt32 (consistent with older tables) - Update execution_engine_new_payload ORDER BY to use block_number first * fix: partition execution_engine_new_payload by block_number Consistent with execution_transaction and the ORDER BY change. * add execution version * feat: add RPC_SNOOPER module for execution engine event source

feat: add RPC_SNOOPER module for execution engine event source

2026-01-14

feat: Add Execution Engine tables

2025-12-19

feat(ethstats): add ethstats CLI subcommand with per-node credential forwarding Add a new ethstats mode that acts as an ethstats server, receiving data from execution clients and forwarding it to configured output sinks. Key features: - WebSocket server compatible with standard ethstats protocol - Per-node credential forwarding: nodes connect with nodename:base64(user:pass) allowing multiple nodes to share credentials with unique display names - GeoIP enrichment support - Authorization system with user/group filtering - Protobuf event types for blocks, stats, pending, latency, and new payloads Also refactors auth and geoip packages to be shared across server components.

2025-12-09

chore: renaming

feat: add exec client impl + ver

feat: engine_get_blobs

2025-12-03

feat: engine_api_new_payload

2025-11-07

feat(sentry): Add new event EXECUTION_STATE_SIZE (#683) * Initial execution state size event * feat: Update event name to execution-state-size * test(execution): add comprehensive EL client version parsing tests - Add unit tests for all 5 EL implementations (Geth, Erigon, Nethermind, Besu, Reth) - Verify version parsing handles various formats (with/without 'v' prefix, different commit hash separators) - Update parseClientVersion documentation with real-world examples from production - All tests passing for current production EL clients * fix(vector): use standard event date_time parsing for execution_state_size - Remove unnecessary if exists() check - Remove fallback to now() which hides parsing errors - Use standard pattern consistent with other events - Properly propagate errors via .error field * revert(migrations): remove extraneous blank line - Remove accidental blank line added in 040_canonical_beacon_validators_split.up.sql - Restore file to previous formatting * refactor(types): align field spacing in DuplicateCache and event types for consistency * feat(db): add updated_date_time to execution_state_size table for better data management This change introduces an `updated_date_time` field in the Vector transform and ClickHouse table schema to track when records were last updated, enabling the use of `ReplicatedReplacingMergeTree` for deduplication based on this timestamp.

2025-10-20

feat(libp2p): Add rpc_custody_probe event (#674) * feat(libp2p): Add rpc_custody_probe event * custody probe fixes * update event name * refactor(event_rpc): use event timestamp instead of job start for request time calculation * fix(libp2p): Fix linting issues in custody probe event handling - Fix StartDateTime calculation for wallclock slots and epochs to use proper TimeWindow().Start() methods - Improve peer ID validation to check for nil before accessing value Co-authored-by: Sam Calder-Mason <samcm@users.noreply.github.com> * feat(proto): add JobStartTimestamp conversion to DataColumnCustodyProbe from trace event payload --------- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> Co-authored-by: Sam Calder-Mason <samcm@users.noreply.github.com>

2025-10-01

feat(metrics): add earliest_available_slot and custody_group_count to libp2p metrics This commit introduces two new fields to track more detailed peer RPC information: - `earliest_available_slot` in `Status` for RPC responses. - `custody_group_count` in `Metadata`. This requires updating Vector configuration, ClickHouse migrations (up/down), and the Go protobuf definitions to correctly ingest and store this new data.

Merge branch 'master' into feat/libp2p-handle-metadata-status-direction

2025-09-30

feat: replace kzg_commitments with kzg_commitments_count in data_column_sidecar Replace the large kzg_commitments array with a lightweight kzg_commitments_count field to reduce storage and bandwidth requirements. Changes: - Add migration 077 to remove kzg_commitments column and add kzg_commitments_count (UInt32) - Deprecate kzg_commitments field in EventDataColumnSidecar proto - Add kzg_commitments_count field to EventDataColumnSidecar proto - Update sentry to populate only kzg_commitments_count - Remove unused convertKZGCommitments helper method - Regenerate proto files

2025-09-29

feat(libp2p): Add direction

2025-09-24

feat: add block root field to libp2p gossipsub blob and data column sidecars - Add beacon_block_root column to ClickHouse tables for blob and data column sidecars - Update proto definitions to include block_root field in BlobSidecar and DataColumnSidecar - Update tests to handle new block_root field - Add .env file with ClickHouse version configuration

2025-08-25

feat(relay-monitor): Add persistence and backfill (#625) * feat(relaymonitor): add backfill functionality for historical slot data This commit introduces a new backfill feature that allows the relay monitor to fetch historical slot data from relays. This is useful for ensuring that all relevant data is captured, even if the monitor was not running during specific periods. The backfill functionality can be configured in the `relay_monitor.yaml` file, allowing users to specify: - Whether backfilling is enabled. - The target epoch or fork to backfill up to. - Rate limiting parameters such as requests per second, slots per request, and delay between relays. The `BackfillManager` has been added to handle the backfill process, including calculating the target slot, managing rate limiting, and iterating through historical slots to fetch data from each configured relay. * feat(config): add coordinator configuration and update backfill options Introduces configuration for a coordinator, enabling persistence and coordination with an xatu server. Also refactors backfill configuration to simplify options and improve clarity. Adds new database migrations for the `relay_monitor_location` table to support coordinator functionality. feat(proto): update coordinator proto with new types and fields This commit introduces several new types and fields to the coordinator proto definition, enhancing its capabilities for node status tracking, discovery, and coordination. Key changes include: - Added `RelayMonitorType` enum for distinguishing different relay monitoring types. - Introduced `RelayMonitorSlotMarker` to track current and backfill slots for relay monitoring. - Added `RelayMonitorLocationBidTrace` and `RelayMonitorLocationPayloadDelivered` for specific relay monitoring data. - Introduced `RelayMonitorLocation` to encapsulate various relay monitoring details, including network, client, relay names, type, and specific data. - Added `GetRelayMonitorLocationRequest`, `GetRelayMonitorLocationResponse`, `UpsertRelayMonitorLocationRequest`, and `UpsertRelayMonitorLocationResponse` for managing relay monitor locations. - Updated `ExecutionNodeStatus_Capability` and `ExecutionNodeStatus_ForkID` with more precise field definitions. - Refined existing message structures like `CannonLocation` and its associated types to include `BackfillingCheckpointMarker` and other relevant fields for improved backfilling and location tracking. - Added `GetCannonLocationRequest`, `GetCannonLocationResponse`, `UpsertCannonLocationRequest`, and `UpsertCannonLocationResponse` for managing cannon locations. - Updated proto generation information with new versions of `protoc-gen-go` and `protoc`. refactor(proto): update coordinator.pb.go with new proto definitions This commit updates the generated Go protobuf definitions for the coordinator service. The changes include: - Renaming the `port` variable to `PORT` for better semantic clarity and consistency with common Go practices. - Adding support for the `PORT` environment variable, allowing the server to be configured with a dynamic port. These changes improve the maintainability and flexibility of the server configuration. feat(proto): update coordinator proto definitions This commit updates the protobuf definitions for the coordinator service. The changes include: - Adding new message types for relay monitor locations and requests. - Expanding the `CannonLocation` enum with new types for various Ethereum beacon chain events. - Introducing the `RelayMonitorType` enum. - Updating existing message structures and field definitions. - Adjusting the service method signatures to accommodate new functionalities. refactor(proto): remove unused proto message exporters The `protoimpl.UnsafeEnabled` block containing message exporters was removed. These exporters are automatically generated by the protobuf compiler and are not needed for the functionality of the code. Removing them cleans up the generated code and reduces its size. feat(proto): add relay monitor location tracking to coordinator service feat(relaymonitor): implement backfill functionality using coordinator for location tracking feat(coordinator): add client and config for interacting with the coordinator service feat(persistence): add relay monitor location persistence layer feat(server): implement relay monitor location gRPC endpoints in the coordinator service * feat: refactor relay-monitor consistency system with shared rate limiting Major improvements to the relay-monitor consistency/backfill system: - Renamed "backfill" to "consistency" to better reflect dual functionality - Split into two independent processes: backfill (works backwards) and forward fill (catches up) - Implemented shared rate limiter per relay (no more wasted capacity) - Forward fill gets priority through shorter timeout on rate limiter - Removed burst configuration (hardcoded to 1 to prevent relay bans) - Added metrics for monitoring lag and current slot - Removed fallback logic that masked configuration errors - Support for sub-1 request/second rates (e.g., 0.5 = 1 req/2s) Breaking changes: - Configuration structure changed from "backfill" to "consistency" - Removed RateLimitBurst configuration (now hardcoded to 1) 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * fix(relay): improve error handling and logging for API requests This commit refactors the `GetProposerPayloadDelivered` and `GetValidatorRegistrations` functions to provide more robust error handling and logging. Specifically, it: - Ensures that API failures are consistently recorded by incrementing metrics before returning an error. - Adds more specific error messages to indicate the cause of failure (e.g., request execution, unexpected status code, response decoding). - Closes the response body in all error scenarios to prevent resource leaks. These changes aim to make it easier to diagnose and resolve issues related to API communication with the relay. * refactor: remove deprecated backfill_slot field from relay monitor The backfill_slot field in RelayMonitorSlotMarker was deprecated and no longer used after implementing process isolation via relay name suffixes. Both backfill and forward fill processes now use only the current_slot field with separate database records identified by ":backfill" and ":forward_fill" suffixes. This change: - Removes backfill_slot field from RelayMonitorSlotMarker proto message - Removes references to backfill_slot from both iterator implementations - Simplifies the code by removing unused compatibility fields 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: start forward fill from current wallclock slot When forward fill iterator has no existing location, it should start from the current wallclock slot instead of slot 0. This ensures forward fill begins processing from the most recent slots rather than unnecessarily starting from genesis. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * fix(relaymonitor): reduce default rate limit per relay to 0.5 and check every duration to 1s to prevent relay bans * fix: resolve linting issues for CI - Add nolint directives for nilnil errors (nil slot indicates no work) - Add nolint directives for gosec integer overflow warnings - Update deprecated golang.org/x/exp/rand to math/rand/v2 - Fix rand.Intn to rand.IntN for math/rand/v2 compatibility - Fix import formatting with goimports 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>

2025-08-21

feat(libp2p): heartbeat (#629) * feat(libp2p): heartbeat * feat(libp2p): rename heartbeat event to synthetic heartbeat Rename LIBP2P_TRACE_HEARTBEAT to LIBP2P_TRACE_SYNTHETIC_HEARTBEAT across the entire codebase to better reflect the nature of this synthetic event. Changes: - Updated proto definitions and regenerated protobuf files - Renamed trace_heartbeat.go to trace_synthetic_heartbeat.go - Updated SQL migration files and table names - Updated Docker Compose vector configurations - Updated all Go code references and type names 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor(clickhouse): remove unique_key from libp2p_synthetic_heartbeat table and use a distributed key for better performance The `unique_key` column was redundant as it was generated using `seahash(.event.id)`. This change removes the `unique_key` column from the `libp2p_synthetic_heartbeat_local` table and updates the `libp2p_synthetic_heartbeat` distributed table to use a composite key based on relevant columns for sharding. This aims to improve query performance and reduce data redundancy. Additionally, the `libp2p-trace-heartbeat` service name in `docker-compose.yml` has been updated to `libp2p-trace-synthetic-heartbeat` to align with the table name change. * feat(event_catergorizer): add synthetic heartbeat event to categorizer This commit adds the `Event_LIBP2P_TRACE_SYNTHETIC_HEARTBEAT` event to the `EventCategorizer`. This event is important for monitoring the health and connectivity of the libp2p network. By categorizing this event, we can ensure that it is properly logged and analyzed, providing valuable insights into the network's performance. * refactor(event): rename HEARTBEAT to SYNTHETIC_HEARTBEAT for clarity * cook * refactor(vector): improve agent version parsing and storage for libp2p heartbeats This commit refactors the agent version parsing logic in the vector configuration and updates the ClickHouse schema to store more granular details about the remote agent. Previously, only a single `agent_version` string was stored. This change extracts and stores the agent's implementation, full version, major, minor, patch versions, and platform separately. This allows for more detailed analysis and filtering of peer data based on specific agent versions or implementations. The `log` statement in the vector transform was also removed as it was redundant. * test(event_categorizer_test.go): update expected event counts from 27 to 28 and group D minimum from 9 to 10 * test(event_categorizer_test.go): correct expected event count in assertion message from 27 to 28 test(event_categorizer_test.go): add missing synthetic heartbeat event to completeness check * ci(test.yaml): remove test annotation step and switch to verbose output - Replace JSON output with human-readable verbose test output - Comment out annotation action to simplify CI pipeline * ci(test.yaml): switch test output to JSON and re-enable annotations - Change `go test -v` to `go test -json` to produce machine-readable output. - Pipe results to `tee test.json` so the file is available for the annotation step. - Uncomment the Annotate tests step so failed tests are surfaced in the PR checks. --------- Co-authored-by: Sam Calder-Mason <sam@puritydev.io> Co-authored-by: Claude <noreply@anthropic.com>

2025-08-20

feat: remove deprecated Blockprint support from xatu-cannon (#628) Blockprint has been deprecated and is no longer supported. This commit removes all Blockprint-related code from xatu-cannon while maintaining proto backward compatibility. Changes: - Remove Blockprint deriver from xatu-cannon initialization - Remove Blockprint configuration from deriver config - Remove Blockprint event handler from xatu-server - Remove Blockprint from Vector kafka/clickhouse configurations - Remove Blockprint topic from docker-compose - Add deprecation comments to proto files (keeping definitions for compatibility) - ClickHouse tables remain unchanged (no data flow) BREAKING CHANGE: Blockprint events are no longer processed by xatu-cannon 🤖 Generated with [Claude Code](https://claude.ai/code) Co-authored-by: Claude <noreply@anthropic.com>

2025-08-07

add kzg count field

feat: add libp2p data column sidecar support - Add protobuf definitions for data column sidecar messages - Create database migration for data column sidecar table - Implement trace gossipsub data column sidecar event handler - Update vector configurations for data column sidecar processing

2025-07-31

feat: add sidecar module proto

2025-07-25

feat: update to go-ethereum v1.16.1 with ETH/69 protocol support - Update go-ethereum from v1.15.11 to v1.16.1 - Update ethcore to latest version supporting ETH/69 - Replace deprecated Status.Head with Status.LatestBlockHash - Mark total_difficulty field as deprecated in protobuf - Add testcontainers dependencies for integration tests

feat(coordinator.proto): add CreateConsensusNodeRecordStatuses RPC for bulk status creation feat(persistence): add BulkInsertNodeRecordConsensus for efficient consensus record ingestion Introduce BulkInsertNodeRecordConsensus to insert many consensus records in a single transaction, batching 500-row chunks to stay within PostgreSQL parameter limits. Includes integration tests that verify correct insertion, empty-slice handling, foreign-key violations, and 600-row batching. feat(coordinator): add CreateConsensusNodeRecordStatuses RPC for bulk status updates Add CreateConsensusNodeRecordStatuses endpoint that accepts a list of consensus statuses, bulk-inserts the corresponding consensus records via BulkInsertNodeRecordConsensus, and updates the parent node records.

2025-07-24

Merge branch 'master' into feat/discovery-cl

2025-07-17

feat(proto): Add TYSM client type (#598) * feat(proto): add new module name `TYPES` to `ModuleName` enum * refactor(proto): rename ModuleName_TYPES to ModuleName_TYSM for brevity

2025-07-16

feat(contributoor): introduce contributoor service for configuration management This commit introduces the new `contributoor` service, designed to provide configuration data to clients. This service allows for dynamic configuration of beacon subscriptions and other settings, supporting global, user-specific, and network-specific overrides. Key changes include: - **New Protobuf Definitions**: Added `contributoor.proto` and its generated Go files (`contributoor.pb.go`, `contributoor_grpc.pb.go`) to define the `GetConfiguration` RPC and associated message types for configuration data. - **Service Implementation**: Implemented the `contributoor` gRPC service client (`pkg/server/service/contributoor/contributoor.go`) which handles `GetConfiguration` requests. - **Configuration Structure**: Defined `Config` and related structures (`BootConfiguration`, `GlobalConfig`, `UserConfig`, `NetworkConfig`, `BeaconSubscriptionsConfig`, `AttestationConfig`) in `pkg/server/service/contributoor/config.go` to manage the various levels of configuration. - **Authentication**: Integrated a basic authentication mechanism (`pkg/server/service/contributoor/auth/`) to secure the configuration endpoint and filter user-specific data based on the authenticated user. - **Default Configuration**: Provided default configuration values for beacon subscription topics and attestation subnets in `pkg/server/service/contributoor/config_default.go`. - **Service Integration**: Updated `pkg/server/service/config.go` and `pkg/server/service/service.go` to include the `contributoor` service in the overall application configuration and service creation flow. This service enables clients to fetch their operational parameters from a centralized source, facilitating easier management and deployment of monitoring agents.

2025-07-14

chore(baseline): merge master

feat: add has_ipv6 to consensus node record

chore(baseline): merge master

feat: add has_ipv6 to execution node record

Merge pull request #580 from ethpandaops/feat/discovery-cl-matty-pt13 feat(noderecord): add TCP, UDP, and QUIC port fields to node record consensus

feat: fulu (#501) * feat: fulu * build: update go-eth2-client dependency to a newer version * feat: implement data column sidecar event support for PeerDAS (EIP-7594) This commit implements complete end-to-end support for data column sidecar events in xatu, enabling PeerDAS (EIP-7594) data availability sampling. - Add EventDataColumnSidecar protobuf message with slot, index, block_root, kzg_commitments - Add BEACON_API_ETH_V1_EVENTS_DATA_COLUMN_SIDECAR event enum (58) - Add DecoratedEvent oneof field for data column sidecar events - Add AdditionalEthV1EventsDataColumnSidecarData metadata structure - Implement sentry-side event handler with decoration and duplicate detection - Implement server-side event handler with validation and routing - Add TTL cache for duplicate detection following existing patterns - Add event type registration and routing in server event ingester - Add ClickHouse migration 058 with optimized schema for data column sidecar events - Include proper indexing, partitioning, and metadata columns - Support for kzg_commitments array and column_index fields - Update Vector configuration with routing, transformation, and ClickHouse sink - Add data column sidecar formatter for ClickHouse ingestion - Support multi-sink outputs (Kafka, HTTP, stdout, xatu) - Update beacon package dependency to v0.53.0 with DataColumnSidecarEvent support - Prepare OnDataColumnSidecar subscription (commented until beacon package update) - Full type safety and end-to-end compilation verified - BEACON_PACKAGE_PATCH.md: Required patch for ethpandaops/beacon package - DATA_COLUMN_SIDECAR_SIMULATION.md: Complete implementation status and usage Once the beacon package adds OnDataColumnSidecar method: 1. Uncomment sentry integration (lines 625-652 in sentry.go) 2. Run ClickHouse migration 058 3. Deploy Vector configuration 4. Start collecting PeerDAS data column sidecar events This implementation is production-ready and follows xatu's established patterns for high-performance event processing and data pipeline integration. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: correct data column sidecar field mappings and migration structure - Fix Vector field mappings: .column_index (not .blob_index), .kzg_commitments (not .kzg_commitment) - Remove non-existent .versioned_hash field from data column sidecar - Include meta_client_clock_drift in CREATE TABLE instead of ALTER TABLE 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Peerdas * add grafana host name flag * fix migrations * merge master * Delete BEACON_PACKAGE_PATCH.md Signed-off-by: Sam Calder-Mason <sam.calder-mason@ethereum.org> * build(deps): update go.mod and go.sum to update dependencies Updates various Go module dependencies to their latest versions, including: - github.com/attestantio/go-eth2-client to support EIP-7805 spec - github.com/ethpandaops/beacon to v0.63.0 - github.com/ethpandaops/ethcore to v0.0.0-20250714031100-d656d160cf25 - github.com/jellydator/ttlcache/v3 to v3.4.0 - github.com/libp2p/go-libp2p to v0.41.1 - github.com/multiformats/go-multiaddr to v0.16.0 - golang.org/x/exp, golang.org/x/sync, golang.org/x/crypto, golang.org/x/net, golang.org/x/sys, golang.org/x/term, golang.org/x/text, golang.org/x/tools - Various indirect dependencies for bug fixes and performance improvements. * docs(go.mod): update comment to reflect correct pull request number for go-eth2-client fork * feat: implement data column sidecar event support for PeerDAS (EIP-7594) (#539) * chore: remove unused create-plan.md file * fix(cannon): Fix chain id deriving * fix(cannon): Fix chain id deriving * feat: implement data column sidecar event support for PeerDAS (EIP-7594) This commit implements complete end-to-end support for data column sidecar events in xatu, enabling PeerDAS (EIP-7594) data availability sampling. ## Features Added ### Core Infrastructure - Add EventDataColumnSidecar protobuf message with slot, index, block_root, kzg_commitments - Add BEACON_API_ETH_V1_EVENTS_DATA_COLUMN_SIDECAR event enum (58) - Add DecoratedEvent oneof field for data column sidecar events - Add AdditionalEthV1EventsDataColumnSidecarData metadata structure ### Event Processing - Implement sentry-side event handler with decoration and duplicate detection - Implement server-side event handler with validation and routing - Add TTL cache for duplicate detection following existing patterns - Add event type registration and routing in server event ingester ### Database Layer - Add ClickHouse migration 058 with optimized schema for data column sidecar events - Include proper indexing, partitioning, and metadata columns - Support for kzg_commitments array and column_index fields ### Data Pipeline - Update Vector configuration with routing, transformation, and ClickHouse sink - Add data column sidecar formatter for ClickHouse ingestion - Support multi-sink outputs (Kafka, HTTP, stdout, xatu) ### Integration - Update beacon package dependency to v0.53.0 with DataColumnSidecarEvent support - Prepare OnDataColumnSidecar subscription (commented until beacon package update) - Full type safety and end-to-end compilation verified ## Documentation - BEACON_PACKAGE_PATCH.md: Required patch for ethpandaops/beacon package - DATA_COLUMN_SIDECAR_SIMULATION.md: Complete implementation status and usage ## Activation Once the beacon package adds OnDataColumnSidecar method: 1. Uncomment sentry integration (lines 625-652 in sentry.go) 2. Run ClickHouse migration 058 3. Deploy Vector configuration 4. Start collecting PeerDAS data column sidecar events This implementation is production-ready and follows xatu's established patterns for high-performance event processing and data pipeline integration. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: correct data column sidecar field mappings and migration structure - Fix Vector field mappings: .column_index (not .blob_index), .kzg_commitments (not .kzg_commitment) - Remove non-existent .versioned_hash field from data column sidecar - Include meta_client_clock_drift in CREATE TABLE instead of ALTER TABLE 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * merge * Remove duplicate AdditionalEthV1EventsDataColumnSidecarData definition * migrations * fix: update go-eth2-client and beacon module dependencies Updates the `go-eth2-client` and `beacon` module dependencies to their latest versions. This includes adjusting the `go-eth2-client` replace directive to point to `github.com/samcm/go-eth2-client` and updating the version of `github.com/ethpandaops/beacon`. This also includes a minor refactor to `EventsDataColumnSidecar` to align with the updated `go-eth2-client` API, specifically changing the type of `event` and introducing a `convertKZGCommitments` method to handle the new KZG commitment format. * feat: add data column sidecar event support Removes the `BEACON_PACKAGE_PATCH.md` file as it's no longer needed. Adds a new Kafka topic and ClickHouse tables for `data_column_sidecar` events. Updates `docker-compose.yml` to include the new Kafka topic. * build: update go-eth2-client dependency to samcm's fork --------- Co-authored-by: Claude <noreply@anthropic.com> * fix(beacon): add type assertion and error handling for FuluBlock casting --------- Signed-off-by: Sam Calder-Mason <sam.calder-mason@ethereum.org> Co-authored-by: Andrew Davis <1709934+Savid@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Matty Evans <mattevansnz@gmail.com>

feat: yeet client_id on execution node records

feat: implement data column sidecar event support for PeerDAS (EIP-7594) (#539) * chore: remove unused create-plan.md file * fix(cannon): Fix chain id deriving * fix(cannon): Fix chain id deriving * feat: implement data column sidecar event support for PeerDAS (EIP-7594) This commit implements complete end-to-end support for data column sidecar events in xatu, enabling PeerDAS (EIP-7594) data availability sampling. ## Features Added ### Core Infrastructure - Add EventDataColumnSidecar protobuf message with slot, index, block_root, kzg_commitments - Add BEACON_API_ETH_V1_EVENTS_DATA_COLUMN_SIDECAR event enum (58) - Add DecoratedEvent oneof field for data column sidecar events - Add AdditionalEthV1EventsDataColumnSidecarData metadata structure ### Event Processing - Implement sentry-side event handler with decoration and duplicate detection - Implement server-side event handler with validation and routing - Add TTL cache for duplicate detection following existing patterns - Add event type registration and routing in server event ingester ### Database Layer - Add ClickHouse migration 058 with optimized schema for data column sidecar events - Include proper indexing, partitioning, and metadata columns - Support for kzg_commitments array and column_index fields ### Data Pipeline - Update Vector configuration with routing, transformation, and ClickHouse sink - Add data column sidecar formatter for ClickHouse ingestion - Support multi-sink outputs (Kafka, HTTP, stdout, xatu) ### Integration - Update beacon package dependency to v0.53.0 with DataColumnSidecarEvent support - Prepare OnDataColumnSidecar subscription (commented until beacon package update) - Full type safety and end-to-end compilation verified ## Documentation - BEACON_PACKAGE_PATCH.md: Required patch for ethpandaops/beacon package - DATA_COLUMN_SIDECAR_SIMULATION.md: Complete implementation status and usage ## Activation Once the beacon package adds OnDataColumnSidecar method: 1. Uncomment sentry integration (lines 625-652 in sentry.go) 2. Run ClickHouse migration 058 3. Deploy Vector configuration 4. Start collecting PeerDAS data column sidecar events This implementation is production-ready and follows xatu's established patterns for high-performance event processing and data pipeline integration. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: correct data column sidecar field mappings and migration structure - Fix Vector field mappings: .column_index (not .blob_index), .kzg_commitments (not .kzg_commitment) - Remove non-existent .versioned_hash field from data column sidecar - Include meta_client_clock_drift in CREATE TABLE instead of ALTER TABLE 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * merge * Remove duplicate AdditionalEthV1EventsDataColumnSidecarData definition * migrations * fix: update go-eth2-client and beacon module dependencies Updates the `go-eth2-client` and `beacon` module dependencies to their latest versions. This includes adjusting the `go-eth2-client` replace directive to point to `github.com/samcm/go-eth2-client` and updating the version of `github.com/ethpandaops/beacon`. This also includes a minor refactor to `EventsDataColumnSidecar` to align with the updated `go-eth2-client` API, specifically changing the type of `event` and introducing a `convertKZGCommitments` method to handle the new KZG commitment format. * feat: add data column sidecar event support Removes the `BEACON_PACKAGE_PATCH.md` file as it's no longer needed. Adds a new Kafka topic and ClickHouse tables for `data_column_sidecar` events. Updates `docker-compose.yml` to include the new Kafka topic. * build: update go-eth2-client dependency to samcm's fork --------- Co-authored-by: Claude <noreply@anthropic.com>

feat(noderecord): add TCP, UDP, and QUIC port fields to node record consensus The `node_record_consensus` table in ClickHouse now includes `tcp`, `udp`, and `quic` columns to store port information from the ENR. The `Consensus` protobuf message has been updated to include `tcp`, `udp`, and `quic` fields. The `AppendServerMeta` function in the `noderecord/consensus.go` service now extracts and populates these port fields from the ENR. This change allows for more detailed network information to be stored and analyzed for consensus nodes. build(deps): update github.com/ethpandaops/beacon to v0.63.0 build(deps): update github.com/ethpandaops/ethcore to 0.0.0-20250714043240-99aba39e37eb

feat: add geo, ip, ports to execution node record

feat(discovery): implement finalized_epoch_start_date_time and head_slot_start_date_time

merge master

2025-07-13

feat(discovery): geo data for consensus node records

2025-07-11

feat: add AggregateAndProof gossipsub event support (#574) * feat: add AggregateAndProof gossipsub event support Add monitoring support for SignedAggregateAttestationAndProof events in libp2p gossipsub networks with full pipeline from client to ClickHouse storage. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor: simplify aggregate and proof event handling Remove unnecessary double switch statement now that we handle the correct TraceEvent types directly. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * feat(clmimicry): implement dynamic committee index derivation for Electra aggregates - Add deriveCommitteeIndexFromBits helper function to extract committee index from committee bits - Update Electra handler to derive committee index dynamically from committee_bits field - Maintain Phase 0 handler using direct committee index access - Ensure both approaches populate same protobuf field for Vector compatibility - Support Ethereum aggregator constraint of one committee per aggregation over gossip 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * feat(proto): add V2 protobuf types for AggregateAndProof events - Add AggregateAttestationAndProofV2 and SignedAggregateAttestationAndProofV2 types - Remove selection_proof and signature fields for V2 types following consensus spec - Update ClickHouse schema to remove unused selection_proof and signature columns - Update Vector configuration to use new aggregator_index field path - Support proper JSON serialization with wrapped UInt64Value types 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>

2025-07-10

feat(consensus): adds nfd. supports devnets. readys for fulu

2025-07-08

refactor(discovery): rename csc to cgc, add peer_id + node_id to consensus node table

2025-06-12

feat: implement data column sidecar event support for PeerDAS (EIP-7594) This commit implements complete end-to-end support for data column sidecar events in xatu, enabling PeerDAS (EIP-7594) data availability sampling. - Add EventDataColumnSidecar protobuf message with slot, index, block_root, kzg_commitments - Add BEACON_API_ETH_V1_EVENTS_DATA_COLUMN_SIDECAR event enum (58) - Add DecoratedEvent oneof field for data column sidecar events - Add AdditionalEthV1EventsDataColumnSidecarData metadata structure - Implement sentry-side event handler with decoration and duplicate detection - Implement server-side event handler with validation and routing - Add TTL cache for duplicate detection following existing patterns - Add event type registration and routing in server event ingester - Add ClickHouse migration 058 with optimized schema for data column sidecar events - Include proper indexing, partitioning, and metadata columns - Support for kzg_commitments array and column_index fields - Update Vector configuration with routing, transformation, and ClickHouse sink - Add data column sidecar formatter for ClickHouse ingestion - Support multi-sink outputs (Kafka, HTTP, stdout, xatu) - Update beacon package dependency to v0.53.0 with DataColumnSidecarEvent support - Prepare OnDataColumnSidecar subscription (commented until beacon package update) - Full type safety and end-to-end compilation verified - BEACON_PACKAGE_PATCH.md: Required patch for ethpandaops/beacon package - DATA_COLUMN_SIDECAR_SIMULATION.md: Complete implementation status and usage Once the beacon package adds OnDataColumnSidecar method: 1. Uncomment sentry integration (lines 625-652 in sentry.go) 2. Run ClickHouse migration 058 3. Deploy Vector configuration 4. Start collecting PeerDAS data column sidecar events This implementation is production-ready and follows xatu's established patterns for high-performance event processing and data pipeline integration. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

2025-06-10

Merge branch 'master' into release/fulu

2025-06-06

merge master

2025-05-28

feat(libp2p): adds trace events for rpc meta and sub

2025-05-26

refactor(vector): improve RPC meta event processing and unique key generation feat(vector): add root_event_id to RPC meta events for better correlation feat(vector): extract topic parts for RPC meta control graft and prune events feat(vector): add original index to filtered RPC meta messages feat(vector): add control index to RPC meta control events feat(vector): add peer index to RPC meta control prune events refactor(clmimicry): pass root event ID down to RPC meta parsing functions refactor(clmimicry): update ShouldTraceRPCMetaMessages to return original index refactor(clmimicry): update RPC meta control parsing to use FilteredMessageWithIndex refactor(proto): update control message meta items to include root event ID and indices The ControlIHaveMetaItem, ControlIWantMetaItem, ControlGraftMetaItem, ControlPruneMetaItem, and ControlIDontWantMetaItem protobuf messages have been updated to include `root_event_id`, `message_index`, and `control_index` fields. This change is necessary to provide more detailed tracing information for control messages. The `root_event_id` links the control message item back to the original event that triggered it. The `message_index` and `control_index` provide the position of the item within the respective message and control message lists, allowing for precise identification and ordering of events in the trace.

feat(libp2p): rpc control trace events

2025-05-22

fix(libp2p): wire up yaml plumbing for new trace events

refactor(libp2p): added seq_number

refactor(libp2p): seqid to seq_number

feat(libp2p): add SeqID field to RejectMessage trace events Add support for capturing sequence IDs from Hermes TraceEvent payloads in libp2p RejectMessage events. This enhancement improves trace data completeness by including the SeqID field that is provided by the Hermes library. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

2025-05-21

refactor(libp2p): events use map for o(1) lookup over sequential check

2025-05-20

docs: add ClickHouse schema and migration rules for LLMs refactor: move LLM rule files from .cursor/rules to llms/rules docs: add LLM guidelines for ClickHouse migrations directory feat(proto): add new CannonType and CannonLocation for consolidation requests by epoch refactor(proto): add new message type for beacon block consolidation request epoch This commit introduces a new message type `CannonLocationEthV2BeaconBlockConsolidationRequestEpoch` to the protobuf definition. This new message is specifically designed to handle requests for beacon block consolidation based on epoch, providing a more granular and specific request type for this operation. This change improves the clarity and organization of the protobuf messages by creating a dedicated type for this specific request. refactor(proto): rename CreateExecutionNodeRecordStatus to ListStalledExecutionNodeRecords feat(proto): add CannonLocationEthV2BeaconBlockConsolidationRequestEpoch message feat(proto): add new cannon type and location for beacon block consolidation request epoch refactor(docs): update Claude markdown files to point to new llms directory for rules feat(persistence): add marshal and unmarshal support for new cannon location type

feat(cannon): implement consolidation requests iterator for Electra fork This implements end-to-end support for EIP-7251 consolidation requests in Xatu: - Add ConsolidationRequest deriver in Cannon to extract consolidation requests from beacon blocks starting from Electra fork - Implement server event handler for processing consolidation request events - Add protocol buffer integration and event type constants - Create ClickHouse migration (062) for canonical_beacon_block_consolidation_request table - Update Vector configuration for Kafka and ClickHouse routing - Add configuration example in example_cannon.yaml The consolidation request contains source_address, source_pubkey, and target_pubkey fields as defined in EIP-7251, enabling monitoring and analytics of validator consolidation operations on the network. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>