Get desktop application:
View/edit binary Protocol Buffers messages
An event server for receiving application reported events. Auth is via bearer token. Client must ed25519 sign the challenge token returned by Authenticate and present it in the `authorization` header as `Bearer <hex(token)>.<hex(sig)>`.
Generate an authentication challenge token for the client to sign.
Stream of events from client to server, and acknowledgements from server to client. The client authenticates by presenting the signed challenge token in the authorization header.
An event sent from the application to the event server
A nonce provided by the application to identify this event, should be acknowledged back in StreamEventsResponse
The unique ID of the event from this application instance, monotonically increasing from zero; restarts from zero when the validator restarts (scope by AuthenticateRequest.instance_id), may have gaps if events get dropped
The producer's per-link sequence number, monotonically increasing from zero, gaps indicate the producer's link to the event tile overran and dropped events
The wall clock time when the event was recorded, in nanoseconds since the Unix epoch
The actual event
Acknowledgement of an event received by the server
The nonce of the acknowledged event
StreamEvents compressed as one zstd stream per call. Each message is the compressor output at one flush, ending on an event boundary, decompressed, the stream is varint(len) | StreamEventsRequest, repeated. An empty message is a heartbeat.
(message has no fields)
A single accounts-database partition finished compacting. Compaction scans one partition, relocates its still-live account records forward into the next storage tier, and reclaims the partition. Emitted once per partition per compaction pass by the accdb tile. Partitions are large (e.g. 32 GiB) so this is a rare, coarse-grained event rather than a hot-path counter.
Used in:
Index of the partition that was compacted, identifying its slot in the accdb partition pool.
Storage tier the partition was compacted from. Layer 0 is the hot tier that takes live execution writes; higher layers are progressively colder. Records are promoted one tier per compaction.
Storage tier the surviving live records were relocated into, normally src_layer+1, clamped at the coldest tier.
Total bytes of account records walked in the partition this pass, live and dead combined. Equals the partition's written extent at the time compaction started.
Dead bytes in the partition that triggered compaction: space held by account records that had been superseded by newer versions. This is the space compaction reclaims.
Number of still-live account records copied forward into dest_layer this pass.
Total bytes of live account records copied forward into dest_layer this pass. Compare to bytes_scanned for the live fraction of the partition.
Number of records skipped because no live index entry referenced them, i.e. records that were already superseded. Together with accounts_relocated this accounts for every record scanned.
Wall-clock time at which this compaction pass began, in nanoseconds since the Unix epoch.
Wall-clock time at which this compaction pass completed, in nanoseconds since the Unix epoch. Subtract start_time for the pass duration.
The accounts database grew its backing file by one partition. The accounts DB is a single file divided into fixed-size partitions; when a write head runs off the end of the allocated file, a new partition is fallocated and the file grows. Emitted once per grow by the accdb tile. Partitions are large (e.g. 32 GiB) so grows are infrequent and each one is a meaningful step in the database's on-disk footprint.
Used in:
Index of the newly allocated partition. Equals old_partition_max, i.e. it is appended at the end of the previously allocated range.
Index of the partition the write head rotated off of to trigger this grow, i.e. the partition that just filled up. UINT64_MAX when there was no prior write head (the very first partition allocated on this layer).
Storage tier whose write head triggered the grow. Layer 0 is the hot tier fed by live execution writes; higher layers are colder compaction-destination tiers. A layer-0 grow means live ingest outran reclamation; a higher-layer grow means compaction is staging more cold data.
Number of allocated partitions before this grow.
Number of allocated partitions after this grow, always old_partition_max+1.
Size of a single partition in bytes; the increment by which the backing file grows.
Total fallocated size of the accounts database file in bytes after this grow, equal to new_partition_max times partition_sz. Tracks the database's on-disk footprint over time.
Wall time the fallocate call that grew the file by this partition took, in nanoseconds. Slow filesystems can take seconds.
True if the accdb tile grew the file ahead of any write head so a later rotation does not wait on fallocate. layer is then 0 and prior_partition_idx is UINT64_MAX, and neither names a triggering write head.
An admin command completed. Emitted once for every command processed by the running validator, whether it succeeded or failed.
Used in:
The admin command that was processed.
The result of processing the admin command.
The command-specific result when result is custom.
Wall-clock nanosecond unix timestamp at which the command started getting processed.
Wall-clock nanosecond unix timestamp at which the command completed.
Payload version requested by the command. Valid only when has_payload_version is true.
Whether the command payload was large enough to contain payload_version.
Size of the received command payload in bytes.
Command-specific public values serialized as JSON. Unavailable values are omitted. Private key material is never included.
The result of processing the admin command.
Used in:
The command completed successfully.
The running validator does not recognize this command.
The request used an unsupported admin command ABI version.
The request size did not match the admin command ABI.
The command is not supported by this validator configuration.
The command returned a command-specific result described by custom_result.
The admin command that was processed.
Used in:
Change the running validator identity.
Read the running validator identity.
Add an authorized voter key.
Remove every authorized voter key.
Request creation of a snapshot.
Take over the identity specified by the given keypair from the machine currently running it.
An Alpenglow consensus cert we received from a peer or constructed ourselves (see our_cert). Unparseable certs are not included. The row timestamp is when we finished processing the cert; the *_time columns are the phases before it, each running until the next. References: Table 6, Definition 12, Algorithm 1, Lemma 48
Used in:
Which slot the cert is for.
Which block the cert is for (only for fast_final, notar and notar_fallback, all zeros otherwise).
What kind of cert is this.
Which validators' primary votes the cert aggregates (NotarVote for notar, fast_final and notar_fallback; FinalVote for final; SkipVote for skip), as a bitmap over the voter's rank in the slot's epoch: rank r is bit r%8 of byte r/8, bytes through the highest signer.
Fallback signer set (NotarFallbackVote for notar_fallback, SkipFallbackVote for skip), a bitmap by rank like voters. Empty for other kinds.
IPv4 address of the peer that relayed the cert to us; ourselves if our_cert, stored as an IPv4-mapped IPv6 address (all zeros if unknown).
Validator identity of the peer that relayed the cert to us (determined from the QUIC client-side TLS certificate); ourselves if our_cert (all zeros if unknown_peer). Not necessarily the creator of the cert: a node re-broadcasts the first valid cert it obtains (whether from self or others).
Whether we constructed the cert ourselves from the votes we had received, rather than a peer delivering it first. Not every cert is self-made; when it is, we reached the threshold before any peer's copy arrived, i.e. we are running fast.
Who we sent the cert to, indexed by rank like voters: identity and address at each rank, all zeros where we did not send (only if our_cert, empty otherwise).
What we did with the cert. unknown_peer and banned_peer are decided on the connection before the cert is decoded, shred_version_mismatch and unranked_peer after decoding, in that order; the rest are Alpenglow-specific.
When we started BLS verifying the cert (unset if our_cert).
When we started broadcasting the cert to our peers, BLS verification having finished (only if our_cert, 0 otherwise).
When we finished processing the cert, may be any of the above stages including short-circuiting at earlier stages.
The peer at this rank.
Used in:
Validator identity of the peer (all zeros if not sent).
IPv4 address of the peer as an IPv4-mapped IPv6 address (all zeros if not sent).
Port of the peer's Alpenglow QUIC endpoint (0 if not sent).
What kind of cert is this.
Used in:
FinalVote aggregate that sums >= 60% of stake, for slot. There is guaranteed to be one unique notarized block_id Final corresponds to.
NotarVote aggregate that sums >= 80% of stake, for block_id; finalizes it in one round.
NotarVote aggregate that sums >= 60% of stake, for block_id (BlockNotarized; Definition 15).
NotarVote or NotarFallbackVote aggregate that sums >= 60% of stake, for block_id. At most four blocks per slot (Lemma 48).
SkipVote or SkipFallbackVote aggregate that sums >= 60% of stake, for slot.
What we did with the cert. unknown_peer and banned_peer are decided on the connection before the cert is decoded, shred_version_mismatch and unranked_peer after decoding, in that order; the rest are Alpenglow-specific.
Used in:
The connection had no known peer: we cleared its identity while closing it, or accepted it before we knew the epoch's validators.
The peer is banned for an earlier invalid signature or cert; connection closed.
Signed under a different shred version than ours.
The peer has no rank in the slot's epoch.
The slot is behind our finality frontier, or in an earlier epoch we have no stake information for.
The slot is beyond our look-ahead window, or in an epoch we have no stake information for.
We already held a cert of this kind for the slot, or for this block if notar_fallback; we keep one per kind (Definition 13), so we dropped the copy without verifying it. The common case: every node broadcasts each cert it accepts. Recorded for notar certs only.
We already hold a valid cert that supersedes this one, e.g. a fast-finalization cert for the block a notarization cert arrives for; ignored.
The signers' stake is below the threshold (Table 6), a signer rank is outside the epoch, or the aggregate BLS signature is invalid; peer banned.
BLS verified (stake threshold of Table 6 and aggregate BLS signature) and stored as the first cert of its kind for the slot, then broadcast to our peers (Definition 13); see broadcast_start_time and broadcast_to.
An Alpenglow consensus vote we received from a peer or cast ourselves (see our_vote). Unparseable votes are not included. The row timestamp is when we finished processing the vote; the *_time columns are the phases before it, each running until the next. References: Table 6, Definition 12, Algorithm 1, Lemma 48
Used in:
Which slot the vote is for.
Which block the vote is for (only for notar and notar_fallback, all zeros otherwise).
Where the voter ranks in the slot's epoch (only if ranked, 65535 otherwise). Also marks their position in the Cert array
Validator identity of the voter, looked up from voter_rank in the slot's epoch (all zeros if voter_rank is 65535).
What kind of vote is this.
IPv4 address of the peer we received the vote from; ourselves if our_vote, stored as an IPv4-mapped IPv6 address (all zeros if unknown).
Validator identity of the peer we received the vote from (determined from the QUIC client-side TLS certificate); ourselves if our_vote (all zeros if unknown_peer). This is the voter whenever voter_rank is set, since votes are only accepted from their voter.
Whether this is our own vote, which we store like any other; received_from_identity is then our identity.
Why we cast the vote, based on the Alpenglow voting algorithm (not_our_vote if it is not ours). References: Algorithms 1 and 2, Definitions 15, 16 and 17.
Who we sent the vote to, by rank like voters: identity and address at each rank, all zeros where we did not send (only if our_vote, empty otherwise).
What we did with the vote. unknown_peer and banned_peer are decided on the connection before the vote is decoded, shred_version_mismatch and unranked_peer after decoding, in that order; the rest are Alpenglow-specific validation.
Final votes for the slot reached 60% of stake (Table 6). Crossing a quorum triggers BLS verification of the votes (see verify_start_time).
Notar votes for block_id reached 80% of stake (Table 6). Crossing a quorum triggers BLS verification of the votes (see verify_start_time).
Notar votes for block_id reached 60% of stake (Table 6). Crossing a quorum triggers BLS verification of the votes (see verify_start_time).
Notar and notar-fallback votes for block_id reached 60% of stake (Table 6). Crossing a quorum triggers BLS verification of the votes (see verify_start_time).
Skip and skip-fallback votes for the slot reached 60% of stake (Table 6). Crossing a quorum triggers BLS verification of the votes (see verify_start_time).
Notar votes for block_id reached 40% of stake, or 20% with skip votes bringing the total to 60% (Definition 16). Crossing a quorum triggers BLS verification of the votes (see verify_start_time).
Skip votes plus notar votes for every block but the most-voted one reached 40% of stake (Definition 16). Crossing a quorum triggers BLS verification of the votes (see verify_start_time).
When we started aggregating the vote into the running BLS aggregate for its slot, kind and block (only if processing_result is accepted, 0 otherwise).
When we started BLS verifying the aggregate this vote completed (only if this vote brought a tally to a quorum of Table 6 or Definition 16, which is when we verify signatures, 0 otherwise).
When we started broadcasting the vote to our peers (only if our_vote, 0 otherwise).
When we finished processing the vote, may be any of the above stages including short-circuiting at earlier stages.
The peer at this rank.
Used in:
Validator identity of the peer (all zeros if not sent).
IPv4 address of the peer as an IPv4-mapped IPv6 address (all zeros if not sent).
Port of the peer's Alpenglow QUIC endpoint (0 if not sent).
What kind of vote is this.
Used in:
Cast on the slot's first block once its parent is ready at a window start, or once we notarized its parent in the previous slot (Algorithm 2, Definition 15).
Cast once the block we notarized has a notarization cert and we cast no skip or fallback vote in the slot (Algorithm 2, Definition 18). Names only the slot.
Cast for every unvoted slot of the leader window on timeout (Definition 17), or on SafeToNotar or SafeToSkip (Algorithm 1). Names only the slot.
Cast on SafeToNotar. At most three per slot (Definition 12).
Cast on SafeToSkip. Names only the slot.
What we did with the vote. unknown_peer and banned_peer are decided on the connection before the vote is decoded, shred_version_mismatch and unranked_peer after decoding, in that order; the rest are Alpenglow-specific validation.
Used in:
The connection had no known peer: we cleared its identity while closing it, or accepted it before we knew the epoch's validators.
The peer is banned for an earlier invalid signature or cert; connection closed.
Signed under a different shred version than ours.
The peer has no rank in the slot's epoch.
The slot is more than eight slots behind our finality frontier, or in an earlier epoch we have no stake information for.
The slot is beyond our look-ahead window, or in an epoch we have no stake information for.
Conflicts with a vote the peer already cast for the slot (Lemma 22, Definition 12).
We already stored this vote or one that covers it: notar and notar-fallback for one block, skip and skip-fallback (Definition 12); ignored.
We already hold a valid cert for the slot, or block, this vote would count toward; ignored.
Vote is accepted and next step is to aggregate (and broadcast, if our_vote).
Why we cast the vote, based on the Alpenglow voting algorithm (not_our_vote if it is not ours). References: Algorithms 1 and 2, Definitions 15, 16 and 17.
Used in:
Not our vote; see our_vote.
Replay completed the block normally, so we vote notar for it.
We hold the certs that make a block a valid parent for the leader window starting at this slot (a notar, notar-fallback or fast-final cert for it, or it is finalized, and skip certs for every slot in between), so we vote notar for the window's first block we held pending.
We hold a notar cert for the block we voted notar for and cast no skip or fallback vote in the slot, so we vote final for it.
The slot's timeout fired with no vote cast in it, so we vote skip on the entire rest of the window containing this slot.
No block but this one can be fast-finalized in the slot: notar votes for it reach 40% of stake, or 20% with skip votes bringing the total to 60%, and its parent is notar-fallback certified (Definition 16). Having voted skip or notar for another block in the slot, we vote notar-fallback for it and skip every unvoted slot of the window.
No block can be fast-finalized in the slot: skip votes plus notar votes for every block but the most-voted one reach 40% of stake (Definition 16). Having voted notar in the slot, we vote skip-fallback for it and skip every unvoted slot of the window.
Our notar or skip vote sent to the leader of slot+8, who packs the slot's reward cert, apart from the broadcast, and resent every probe timeout until that leader acknowledges it or we replay slot+8.
The leader of slot+8 acknowledged a packet carrying our notar or skip vote, which ends its reward resends. The row timestamp is when the acknowledgement arrived; broadcast_to is empty.
We are trying to cast a notar or skip vote whose reward (slot+8) leader we have no active connection to. The problematic gossip ip and port is in broadcast_to.
A block finished production, or replay on this validator, or was marked dead, or was abandoned before completing. A block gets a row if it was ever replayed or produced (had a bank), or if it finished reception on a fork that had already died or been discarded; a block dropped before reception finished gets no row. Under alpenglow, fork knowledge is much more transient, so blocks that were never replayed due to a parent getting discarded gets no row, even if reception finished. Delivery is best-effort: rows travel a lossy telemetry link, and a large burst of simultaneous rows (mass death of a fork) can drop some. On-chain data is deliberately not part of this event. Reception and repair tallies are best-effort: they restart if repair transiently stopped tracking the block (forest eviction), and on equivocated slots they are slot-scoped — shreds from all observed versions of the slot merge into the same counts on every version's row.
Used in:
Bank sequence number, counting from 1 within a run; the join key to confirmation events. Not meaningful across restarts. 0 means no bank was ever allocated (a block completing reception under a dead or discarded lineage without ever getting a bank), leaving nothing to join on bank_seq; any forward confirmations for such a block join on slot and block_id. The boot bank (bank_seq 1) emits confirmation events but no block_completed row.
Bank pool index the block was tracked under: the identifier log lines print. Instance-local and recycled across blocks within a run; correlate with logs only within one validator instance and time window. Reported whenever a bank existed (any row with nonzero bank_seq). The *_fork_id and cost_tracker_pool_idx columns are debug identifiers with the same scope, reported as held when the row was emitted, and 0 unless the block began replay or production (preparation_begin_time set).
Transaction cache fork handle held by the bank. See bank_idx for scope.
Program cache fork handle held by the bank. See bank_idx for scope.
Account database fork handle held by the bank. See bank_idx for scope.
Vote stakes fork handle held by the bank. See bank_idx for scope.
Deprecated collector overrides fork handle. Always 0.
Stake rewards fork handle held by the bank. See bank_idx for scope.
Epoch credits fork handle held by the bank. See bank_idx for scope.
Stake delegations fork handle held by the bank. See bank_idx for scope.
Cost tracker pool index the bank held: snapshotted just before the freeze releases it on completed rows, read live on dead and abandoned rows; 0 if the block never began replay or production. See bank_idx for scope.
Slot number of the block.
Epoch containing the slot.
Slot number of the parent block. 0 if the parent was no longer locally known when the row was emitted; on genesis-booted clusters, 0 is also the genuine parent slot of blocks chaining directly off genesis.
Finalized root slot when the row was emitted.
Lowest slot still held in memory when the row was emitted.
Highest slot observed in completed FEC sets from the network (own leader blocks excluded) when the row was emitted: the validator's estimate of the cluster tip. Minus slot: how far behind this row was - 0 to 2 in steady state, large during boot catch-up and while racing back after any stall, re-detecting mid-run lag that caught_up cannot. 0 before any shred has been observed.
Whether the validator considered itself caught up to the cluster when the row was emitted. False from boot until replay closes on the network tip (set immediately for genesis bootstrap); never cleared within a run.
Number of forks executing concurrently when the row was emitted. 1 in calm operation; higher values mark fork storms, explaining replay contention, evictions, and prune bursts on the rows that suffered them.
Whether this validator was writing a snapshot when the row was emitted; snapshot creation periodically stalls replay, so timing outliers on these rows are expected. Always false when snapshot creation is unsupported or disabled.
Banks allocated when the row was emitted: executing, awaiting replay, and completed or dead awaiting prune. Bank tracking pressure; blocks are evicted when this reaches the pool bound. fork_width isolates the concurrently-executing component.
Block identifier. For blocks that never completed reception (dead or abandoned early): the merkle root of the latest piece received.
Block id of the parent this block chained from, recovered from the reception chain; disambiguates the parent version when the parent slot equivocated. 0 if the reception chain could not be walked to the parent: reassembly no longer tracked the block, or an abandoned leader block that shredded nothing; parent_slot can still be known on such rows.
Bank hash computed by replaying the block; the state commitment consensus votes on. Rows sharing slot and block_id but differing here indicate state divergence. 0 if the block never froze (dead or abandoned).
Whether the block was ruled invalid. Blocks merely given up on are not dead; see abandoned_reason. State accumulated before the ruling is still reported, so execution fields may be partially filled.
Why the block was ruled invalid; not_dead otherwise. Blocks this validator produced are never dead: they do not go through replay's block checks. A block invalid for several independent reasons records the first one detected locally, which can differ across validators for the same block.
When the block was ruled invalid; 0 if not dead.
Whether this validator gave up on the block before it completed. Independent of dead: an abandoned block was not invalid. Equivalent to abandoned_reason != not_abandoned. State accumulated before abandonment is still reported, so execution fields may be partially filled.
Why this validator gave up on the block before it completed; not_abandoned otherwise. Independent of dead.
When the block was abandoned; 0 if not abandoned.
Whether this validator produced the block as leader.
Network arrival at the shred tile of the block's first shred, stamped before signature verification. Leader blocks: when the first entry batch reached the shredding stage, before any network send. 0 if repair pruned the block before completion.
Network arrival of the shred that made the block contiguous; not necessarily the highest-numbered shred. Minus first_shred_received_time: reception duration, valid only when this is nonzero. Leader blocks: when the final entry batch reached the shredding stage. 0 if the block never became fully contiguous (dead or abandoned mid-reception) or repair pruned it before completion.
First repair request for a specific missing shred (window request); highest-shred and orphan requests do not stamp this, so it can be 0 while their request counts are not. 0 if no window request was sent or repair pruned the block; always 0 for leader blocks.
Arrival of the last repair response matched to an outstanding specific-shred request; shreds from highest-shred and orphan responses do not stamp this. 0 if none was matched or repair pruned the block; always 0 for leader blocks.
When the block's first complete FEC set reached replay, after verification and reassembly. Minus first_shred_received_time: internal pipeline latency from network to replay, valid only when both are nonzero. Leader blocks: when the block's own first FEC set returned to replay through reassembly; follows rather than precedes preparation_begin_time, since production shreds the block as it goes. 0 if no bank was ever allocated, and on abandoned leader blocks whose first FEC set had not returned through reassembly when production was abandoned.
When replay began setting up the block's execution state: cloning from the parent and applying start-of-block updates (epoch boundary processing, fee and rent state). Follows first_fec_set_received_time on replayed blocks; leader blocks reverse that order (see that field). Gap to first_transaction_scheduled_time: setup cost, which for leader blocks includes waiting for pack to schedule the first transaction; valid only when first_transaction_scheduled_time is nonzero. 0 if the block was dropped before setup.
When replay dispatched the block's first transaction to an execution tile. Leader blocks: when an execution tile began the first packed transaction. Execution and verification run from here to last_transaction_finished_time; equal to it if the block executed no transactions or, for leader blocks, if production timing was unavailable (see txn_timing). 0 on dead and abandoned rows if no transaction was ever dispatched.
When the block became fully executed and verified and end-of-block processing began: all execution and signature-verification results returned, the final FEC set received, PoH verified. At the live tip this trails the last execution result by trailing-tick reception and verification; per-transaction results are txn_timing.replayed_time. Leader blocks: when the block was fully produced (final PoH hash received, block id returned through reassembly), trailing the last execution result by the PoH tail and final-entry shredding, most visibly when a limit ended the block early. 0 if the block never finished executing (dead or abandoned).
When end-of-block processing finished and the block froze: state hashing complete, no further changes. Minus last_transaction_finished_time: finalization cost. 0 if the block never completed (dead or abandoned).
The parent's block_completed_time, denormalized to avoid a self-join; for blocks chaining directly off the snapshot or genesis boot bank, when the boot bank finished loading. 0 if the parent's completion was never observed: parent untracked, or parent never froze (dead or abandoned lineage). Slot-time queries (block_completed_time minus this) should filter on caught_up, parent_slot = slot - 1, and a nonzero value here.
Number of FEC sets in the block; for blocks that never completed reception, the number delivered to replay when the row was emitted. 0 if reassembly no longer tracked the block when the row was emitted.
Data shreds received or recovered. Leader blocks: shreds produced. 0 if repair pruned the block before completion.
Parity (recovery) shreds received. Near 0 for leader blocks: only network echoes of the block's own parity shreds land here. 0 if repair pruned the block before completion.
Shreds received over the network via turbine, plus repair deliveries whose provenance could not be verified (failed nonce) and coding shreds arriving in repair responses (nonconformant: repair serves only data shreds). Pieces cleared and re-fetched after a wrong-version detection or eviction count once per delivery, so source tallies can exceed the block's shred counts. Near 0 for leader blocks. 0 if repair pruned the block before completion.
Shreds obtained from peers via repair; nonce-verified deliveries only. Re-fetched pieces count once per delivery (see turbine_shred_count). Always 0 for leader blocks. 0 if repair pruned the block before completion.
Shreds reconstructed locally from parity data. Re-recovered pieces count once per recovery (see turbine_shred_count). Always 0 for leader blocks, which produce rather than recover their shreds. 0 if repair pruned the block before completion.
Whether every FEC set was verified to chain back to the confirmed block identity, i.e. the assembled data matches the block the cluster confirmed. Verification is confirmation-triggered and the value is snapshotted at the block's last received FEC set, so false means it had not run by then, not that it failed; at the live tip confirmation usually arrives later. Always false under alpenglow. True mainly when confirmation precedes reception completing, e.g. during catch-up. See lowest_verified_fec_index.
Lowest FEC set index reached by chain verification as of the block's last received FEC set; the walk anchors at the confirmed block identity and works backwards. 0 means the whole block verified. 4294967295 means no verification information (verification never ran, or repair pruned the block); not a FEC index. Always 0 under alpenglow.
FEC set index of the last piece of the block to complete reception, which with out-of-order repair need not be the block's final index. 4294967295 when the reception statistics are unpopulated; not a FEC index.
Whether the shred marking the block complete was received. False if repair pruned the block before completion.
Whether any of the block's FEC sets was ever quarantined by an equivocation hold: set when a conflicting version of a FEC set was assembled at that position, and inherited by everything chaining onto or arriving during an unresolved conflict, including from ancestor slots; persists after resolution. A lone conflicting shred discarded before reassembly does not register. False if reassembly no longer tracked the block when the row was emitted.
Repair requests re-sent after a response timeout. Always 0 for leader blocks. 0 if repair pruned the block before completion.
Repair responses matched to an outstanding specific-shred request. Shreds delivered by verified highest-shred and orphan responses count in repair_shred_count but are not matched here, so this can be 0 while repair_shred_count is not. Always 0 for leader blocks. 0 if repair pruned the block before completion.
Repair requests for one specific missing shred, by index. Always 0 for leader blocks. 0 if repair pruned the block before completion.
Repair requests for the highest-numbered shred a peer holds, used to learn the block's length. Highest-shred requests seeded at boot, before the block was tracked, bypass this counter. Always 0 for leader blocks. 0 if repair pruned the block before completion.
Repair requests for the block's ancestry when its parent is unknown; a response carries shreds from up to 11 ancestor blocks. Always 0 for leader blocks. 0 if repair pruned the block before completion.
Whether chain verification against the confirmed block identity found a mismatched (wrong-version) FEC held for this block; the piece was cleared and re-repaired, but may have arrived by any route, turbine included. Always false for leader blocks and under alpenglow. False if repair pruned the block before completion.
Compute units used by all transactions, votes included; the vote/non-vote split is not part of this event. 0 if no cost tracker existed for the block (never prepared for replay, or no bank).
New account data allocated, in bytes. 0 if no cost tracker existed for the block.
Block compute unit limit. 0 if no cost tracker existed for the block (never prepared for replay, or no bank).
Deprecated runtime vote compute unit limit. Always 0.
Per-account compute unit limit. 0 if no cost tracker existed for the block.
When replay began block production for the slot, all prerequisites (e.g. parent block identity) in hand. Minus leader_slot_start_time: local production start delay. 0 for blocks this validator did not produce.
When the leader slot was scheduled to begin, estimated locally by projecting slot durations from the reset onto the parent fork; embeds upstream lateness rather than tracking a cluster-absolute timeline. 0 for blocks this validator did not produce.
When block production began. 0 for blocks this validator did not produce.
When block production ended. 0 for blocks this validator did not produce.
Microblocks pack scheduled during production. A microblock whose transactions all failed execution is skipped at PoH mixin and absent from the block, so this can exceed block content. 0 for blocks this validator did not produce.
Compute units pack reserved while building the block, votes included. Reservations are refunded as transactions finish, so this can exceed cost_tracker_block_cost (the charged amount) by refunds unsettled at slot end. 0 for blocks this validator did not produce.
Deprecated (0) since Firedancer v26.11.
Bytes pack's cost model reserved for block content: transaction payloads plus a 48-byte header per microblock, net of rebates settled by slot end. Excludes tick entries; can exceed the block's actual content by rebates unsettled at slot end. A size, not a compute measure. 0 for blocks this validator did not produce.
Why block production stopped; not_leader for blocks this validator did not produce.
Bundled transactions pack scheduled during production, counted whether or not the bundle survived to the block (aborted bundles are skipped at PoH mixin). 0 for blocks this validator did not produce.
Per-transaction timestamps, one entry per transaction in block order. Replayed blocks: stamped by the replay scheduler. Leader blocks: stamped by the production pipeline. Columns exclusive to the other path are 0. Empty on dead and abandoned replay rows; an abandoned leader block reports the transactions mixed into the block before production was abandoned. Otherwise best-effort: empty when no capture slot was free (more than 16 blocks replaying at once) or the production timing table was unavailable. Empty does not mean the block had no transactions.
Whether the block reached replay after a votor event named that block and we repaired that version, rather than through the version received over turbine. Always false before alpenglow, and false for blocks this validator produced as leader. A slot can report both, on separate rows.
Alpenglow ShredForBlockId requests, analogous to legacy window requests. Asks a peer for a specific missing shred, addressed by the block id it belongs to. Always 0 before alpenglow and for leader blocks.
Alpenglow ParentAndFecCount requests, analogous to legacy orphan requests. Asks a peer for the block's parent and its FEC set count. Always 0 before alpenglow and for leader blocks.
Alpenglow FecRoot requests: asks a peer for a FEC set's merkle root for a specified FEC set index. Always 0 before alpenglow and for leader blocks.
Why this validator gave up on the block before it completed; not_abandoned otherwise. Independent of dead.
Used in:
Not abandoned; the block completed or was ruled invalid.
A block on another fork was rooted, discarding this fork before replay finished.
Dropped for lack of room to track in-progress blocks. The row is emitted once the block's in-flight verification has drained without ruling it invalid; a block ruled invalid while draining emits a dead row instead. May be replayed later, producing a second row for the same slot and block id under a new bank_seq; the retry usually completes but can itself end dead, evicted, or pruned. Blocks evicted after freezing already emitted a completed row and emit nothing further. Dedup on slot and block_id, keeping the highest bank_seq.
Production of this leader block was abandoned mid-slot: consensus reset block production onto another fork before the slot completed. Only on blocks this validator produced; pack_end_reason is abandoned on the same row. The partial block never finished shredding, so block_id holds the merkle root of the last piece that had returned through reassembly when production was abandoned (may trail the last piece actually shredded), or 0 if none had.
Prefix of a block we executed before detecting an UpdateParent marker. The prefix execution is abandoned, and the block execution after the UpdateParent marker shows up as new row.
Why the block was ruled invalid; not_dead otherwise. Blocks this validator produced are never dead: they do not go through replay's block checks. A block invalid for several independent reasons records the first one detected locally, which can differ across validators for the same block.
Used in:
Not ruled invalid.
The block went down with its lineage: an ancestor was ruled invalid, so this block cannot be replayed.
Bytes at the head of the stream failed to parse as any structure (transaction, microblock header, or count) within the largest size a valid block allows: malformed content.
Block bytes ended short of the microblocks and transactions declared.
Block did not end on a tick; a transaction entry trailed the final tick. Checked before tick-count verification, so a block Agave rules TooFewTicks can report trailing_entry here.
More microblocks than a valid block can hold.
More hashes since the last tick than hashes per tick allows, detected at tick verification.
A microblock header declared more hashes than fit before the next tick, detected at FEC ingest.
A tick's hash count differs from the block's preceding ticks, detected at FEC ingest.
A tick advanced zero hashes, detected at tick verification because proof-of-history parameters were unknown when the tick parsed.
A tick advanced zero hashes, detected at FEC ingest.
A batch header declared zero microblocks.
More transactions than a valid block can hold.
More ticks than allowed.
Tick hash count did not advance the expected hashes per tick.
Fewer ticks than required.
Proof-of-history hash of a tick did not verify.
Proof-of-history hash of a transaction entry did not verify, detected when the entry's hashing task completed.
Proof-of-history hash of a transaction entry did not verify, detected at FEC ingest when a later FEC set completed the entry's transactions.
A transaction failed signature verification.
A transaction failed a pre-execution check: already processed, expired or unknown recent blockhash, address lookup table resolution failure, account lock limit, compute budget error, unsupported transaction version or sanitization failure, or fee-payer validation failure.
A transaction pushed the block over its compute limit.
A transaction pushed one account over its per-block compute limit.
A vote transaction pushed the block over its vote compute limit.
A transaction pushed the block over its account data size limit.
A transaction referenced the same account more than once.
An Alpenglow block's footer could not be applied or declared a bank hash that did not match execution.
An Alpenglow block had an entry whose hash count was not exactly one.
An Alpenglow block carried an entry batch or footer before any block header.
An Alpenglow block carried more than one block header.
An Alpenglow genesis certificate marker did not immediately follow the block header.
An Alpenglow block carried more than one block footer.
An Alpenglow block carried an entry batch other than the alpentick after its footer.
An Alpenglow block ended without the alpentick directly after its footer.
An Alpenglow block ended without a block footer.
An Alpenglow block started on a UpdateParent marker, or found one after its footer.
An Alpenglow block marker (header, footer, genesis certificate or update parent) failed to parse or had an unknown kind.
The block's transactions declare more writable accounts than a valid block can afford.
An Alpenglow block carried more than one UpdateParent marker.
An Alpenglow block's initial UpdateParent named a parent other than the one replay built it on.
Why block production stopped; not_leader for blocks this validator did not produce.
Used in:
Not produced by this validator.
The leader slot's time window ended, the normal case. Blocks that filled a consensus limit also end here; measure saturation via pack_block_cost and pack_data_bytes against the protocol caps.
Microblock count limit reached.
Production was abandoned mid-slot by a reset onto another fork; only on abandoned leader blocks (abandoned_reason reset). Pack ends the block early and its summary rides on the terminating message, so microblock_count, pack costs, pack timings, and bundle_txn_count reflect what was produced before the abandonment.
Timestamps for one transaction. A zero timestamp means the stage did not occur on this path, was never reached, or its stamp was unavailable.
Used in:
Earliest time attributable to the transaction. Replayed blocks: network arrival of the shred completing the FEC set carrying the transaction; gap to parsed_time is the internal latency to parse. Leader blocks: entry into this validator — TPU network arrival, block engine receipt (bundles), gossip receipt (votes), or creation time (self-originated).
When the scheduler parsed the transaction out of received data, including address lookup table resolution. Gap from received_time covers the pipeline hop, in-order delivery waits, and parse position within the FEC set. 0 for leader blocks.
Replayed blocks: when the scheduler handed the transaction to an execution tile. Leader blocks: when an execution tile began executing it; transactions of one bundle share the stamp.
Replayed blocks: when the scheduler received the execution result, including the return hop and replay's polling delay, not only execution. Leader blocks: when execution finished and results were committed to the block's state.
When the scheduler dispatched signature verification, which runs in parallel with execution. 0 for leader blocks (verified upstream of pack).
When the signature verification result returned to the scheduler. 0 for leader blocks.
When the transaction was mixed into the PoH stream, fixing its position in the block; transactions of one microblock share the stamp. 0 for blocks this validator did not produce.
Two different blocks (different block ids) were observed for the same slot. Emitted once when the validator first detects the equivocation, after both blocks are known.
Used in:
Sequence number of this validator's replayed block, joining to block_completed; zero if there is no local replayed bank (forward-confirmed or shred-proof detection).
The slot that equivocates (has more than one block).
The slot number of the parent block.
The epoch this slot belongs to.
Identifier of the block this validator replayed for the slot.
Identifier of the conflicting block for the same slot; zero if not known (shred-proof detection).
This validator's bank hash for the block it replayed; zero if the block was not replayed locally.
Last microblock header hash of the block this validator replayed; zero if not replayed locally.
Whether this validator produced the equivocating block it replayed.
Whether this validator had voted for its replayed block when the equivocation was detected.
Whether this validator's replayed block had been duplicate confirmed when the equivocation was detected.
Stake in lamports voted on the block this validator replayed at detection; zero if none/unknown. Typically small, as an equivocating block usually did not win confirmation.
Stake in lamports on the conflicting (typically confirmed) block at detection; zero if not known.
Total cluster stake in lamports at detection; the denominator for the stake fields. Zero if not known.
How the equivocation was detected.
How the equivocation was detected.
Used in:
A second, different block for the slot finished replaying locally.
Votes confirmed a block id for the slot that differs from the one this validator replayed.
A duplicate-shred proof showed the slot equivocates; the conflicting block id is not known, so sibling_block_id is zero.
A block that completed reception on this validator or was abandoned and pruned. A block gets a row if it received a shred marked as final in the slot, and received all shreds up to the final shred, or if it was incomplete but collected later during pruning. Tallies are best-effort: on equivocated slots, shreds shared between versions of the slot may merge or double count. Delivery is best-effort: rows travel a lossy telemetry link, so a block_completed row can arrive without its block_received row or vice versa.
Used in:
Slot number of the block.
Block identifier; with slot, the alternate join key to block_completed. For blocks that never completed reception (or were abandoned early), this can be the null hash.
Slot number of the block's parent. If unknown, this will be UINT64_MAX, and likely means that a block header could not be parsed for this slot.
Block identifier of the block's parent. If unknown, this will be the null hash, and likely means that a block header could not be parsed for this slot.
Whether the block was cancelled due to a notar version becoming known. These blocks will only get reported if they are eventually pruned, so if the root does not advance, they will not be reported.
The reason the block was no longer attempted to be repaired.
When the block was cancelled. Only turbine versions are cancelled. Occurs when a votor-driven version of the slot arrives while this version's block id is still unknown, or for invalid turbine blocks. 0 if not cancelled.
Either reached SafeToNotar locally or attached to a NotarFallback or stronger cert, rather than being the version received over turbine. Always false for blocks this validator produced as leader. A slot can report both a notarized and a non-notarized version, on separate rows.
Whether the block was produced as leader.
Whether the validator considered itself caught up to the cluster when the row was emitted. False from boot until repair closes on the network tip (set immediately for genesis bootstrap); otherwise during regular operation is set to True if the highest slot the validator has received completely is no more than one leader window behind its view of the current cluster slot.
Number of FEC sets in the block; for blocks that never completed reception, this could be 0 if the shred with the slot_complete flag never arrived.
Network arrival at the shred tile of the block's first shred, stamped before signature verification. Leader blocks: when the first entry batch reached the shredding stage, before any network send.
Network arrival of the shred that made the block contiguous; not necessarily the highest-numbered shred. Minus first_shred_received_time: reception duration, valid only when this is nonzero. Leader blocks: when the final entry batch reached the shredding stage. 0 if the block never became fully contiguous (pruned or abandoned mid-reception).
First repair request for a specific missing shred (window or ShredForBlockId request); highest-shred, orphan, ParentAndFecCount and FecRoot requests do not stamp this, so it can be 0 while their request counts are not. 0 if no window or ShredForBlockId request was sent; always 0 for leader blocks.
Network arrival at the shred tile of the last repair response matched to an outstanding specific-shred request; other responses do not stamp this. 0 if none was matched; always 0 for leader blocks.
Network arrival at the repair tile of the block's first verified getParentAndFecCount or FecRoot response, including a FecRoot answer for a FEC set already held. 0 if no verified metadata response was received.
Shreds received over the network via turbine, plus repair deliveries whose provenance could not be verified (failed nonce) and coding shreds. Near 0 for leader blocks.
Parity (recovery) shreds received. Near 0 for leader blocks: only network echoes of the block's own parity shreds land here.
Shreds obtained from peers via repair; nonce-verified deliveries only. Re-fetched pieces count once per delivery. Always 0 for leader blocks.
Shreds reconstructed locally from parity data. Always 0 for leader blocks, which produce rather than recover their shreds.
FEC set index of the last piece of the block to complete Reed-Solomon reconstruction, which with out-of-order repair need not be the block's final index. UINT32_MAX if no FEC set was completed.
Whether the shred marking the block complete was received, or it became known through a getParentAndFecCount request.
Whether equivocating shreds were detected on this slot before any notarized version arrived.
Repair responses matched to an outstanding specific-shred request. Shreds delivered by verified highest-shred and orphan responses count in repair_shred_received but are not matched here, so this can be 0 while repair_shred_received is not. Always 0 for leader blocks.
Metadata responses matched to an outstanding getParentAndFecCount request that pass proof-check.
Metadata responses matched to an outstanding FecRoot request that pass proof-check.
Repair requests for one specific missing shred, by index. Always 0 for leader blocks.
Repair requests for the highest-numbered shred a peer holds, used to learn the block's length. Highest-shred requests seeded at boot, before the block was tracked, bypass this counter. Always 0 for leader blocks.
Repair requests for the block's ancestry when its parent is unknown; a response carries shreds from up to 11 ancestor blocks. Always 0 for leader blocks.
Alpenglow ShredForBlockId requests, analogous to legacy window requests. Asks a peer for a specific missing shred, addressed by the block id it belongs to. Always 0 before Alpenglow and for leader blocks.
Alpenglow ParentAndFecCount requests, analogous to legacy orphan requests. Asks a peer for the block's parent and its FEC set count. Always 0 before Alpenglow and for leader blocks.
Alpenglow FecRoot requests. Asks a peer for the merkle root of the FEC set at a given index. Always 0 before Alpenglow and for leader blocks.
Per-FEC-set reception detail, one entry per FEC set in block order: which pieces made up the block, how each arrived, and when. Summed, the counts may reconcile with the block-level shred counts on this row, except in the case of equivocation where FEC sets may be shared between versions of the slot. Best-effort for pruned incomplete blocks: empty does not mean the block had no FEC sets, but that the FEC may have been pruned before it was able to be reported.
The reason the block was no longer attempted to be repaired.
Used in:
Not cancelled.
Received two merkle roots for the same FEC set.
A verified version of the block became known.
A verified version of a verified block's parent became known.
A data shred's parent_off disagreed with the slot's other shreds or with its block header.
Shred 0 did not carry a valid block header.
A data shred's slot complete flag disagreed with the slot's other shreds.
An UpdateParent marker broke a rule: not in the first slot of its leader window, a second UpdateParent, a parent below the root or newer than the header's, a null parent block id, or naming the header's own parent.
Reception detail for one FEC set. A zero timestamp means the stage was never reached or its stamp was unavailable.
Used in:
Merkle root of the FEC set: its identity. The final FEC set's merkle root is the block id.
Index of the FEC set's first data shred within the block; FEC sets are listed in block order, so for a fully received block the indices ascend from 0 without gaps.
Bitmap of data shreds of the FEC set received (vs recovered). Bit j set if the data shred with data index j was received.
Bitmap of parity shreds of the FEC set received (vs recovered). Bit j set if the parity shred with parity index j was received.
Bitmap of data shreds of the FEC set received via repair. Bit j set if the data shred with data index j arrived via repair.
Number of duplicate shreds received for this FEC set; best-effort given upstream deduplication.
Timestamp when the first shred of this FEC set arrived at the shred tile.
Timestamp when the FEC set was complete (assembled).
The source of the final shred that completed the FEC set.
The source of the final shred that completed the FEC set.
Used in:
Received via turbine.
Received via repair.
Produced as our own leader shred.
Reported once when the validator process starts. A wide fingerprint of the host, operating system, kernel, hardware, network, build, and validator configuration at launch.
Used in:
Kernel release string, such as 6.1.0-1202605170043.el9.x86_64.
Full kernel build version string including build date and preemption model.
Processor vulnerabilities for which a kernel mitigation is active; mitigations in force split performance cohorts on otherwise identical hardware. Empty when every mitigation is disabled or the hardware is unaffected.
Short distribution identifier such as ubuntu, rhel, debian, or centos.
Distribution version number, such as 22.04 or 9.4.
Family of distributions this one derives from, such as debian or rhel.
Which system C library the validator is running against.
Version of the system C runtime library, such as 2.34.
The named hardware target the binary was compiled for, which fixes the minimum processor it can run on. Distinct from the live processor it is currently running on.
The list of optional build features requested when the binary was compiled, such as sanitizers or coverage. Empty for a normal production build.
Whether tracked files in the source tree had uncommitted changes when the binary was built, meaning the commit hash alone does not fully describe the code. Untracked files do not count.
Which compiler family produced the binary.
The exact version of the compiler that produced the binary.
When the binary was compiled; distinguishes rebuilds of the same commit, which build_git_dirty makes common on development builds.
Instruction set extensions the binary was compiled to use, such as avx512 or gfni; fixes the minimum processor the binary can run on and which accelerated code paths are compiled in.
Processor architecture of the host.
The maker of the processor.
The full marketing name of the processor.
The processor family number identifying the broad generation of the chip.
The numeric model identifier within the family, distinguishing specific chip designs.
The silicon revision number of the processor, indicating minor hardware fixes within a model.
The currently loaded microcode firmware revision of the processor, important because microcode updates change performance and security behavior.
The total number of logical processors the operating system sees, counting each hardware thread separately.
The number of physical cores across all processor sockets, ignoring hardware threads.
The number of physical processor packages installed in the machine.
Whether simultaneous multithreading (hyperthreading) is currently enabled on the host.
The active frequency-scaling policy, which determines how aggressively the chip clocks up.
The kernel driver controlling processor frequency scaling.
Energy-performance preference in force on the frequency driver; a balance profile under governor performance silently caps sustained clocks.
The kernel driver managing processor idle low-power states.
Name of the deepest processor idle state currently enabled; deeper states add wake-up latency on pinned cores.
Total usable physical memory detected by the kernel at boot, in bytes.
Estimate of memory available for starting new applications without swapping at boot, in bytes.
For each logical processor, the memory locality domain it belongs to; joins tile placements against the locality of network and storage devices to find cross-domain pinning.
Logical processors isolated from the kernel scheduler at boot by the isolcpus= kernel parameter. Isolation by the validator's cpuset partition is not listed here; see cpuset_partition.
Per-module memory inventory: mismatched speeds clock the whole bus down, and a module differing after a swap explains a machine that got slower after maintenance.
Inventory of physical network interfaces, each with its identity, driver, hardware location, link characteristics, queue configuration, and offload settings.
Names of the physical member interfaces that make up the validator's bonded device.
Bonding mode of the validator's bonded device, if bonded; kernel-bypass support and hashing behavior differ by mode.
Index into nic_devices of the interface bound for the kernel-bypass path, the bond master entry when bonded; 255 when the bound interface is not in the inventory.
Whether the kernel just-in-time compiler for packet programs is enabled, which is needed for efficient kernel-bypass networking.
System manufacturer of the physical machine, or the virtualization vendor when virtualized.
Model or product name of the system, or a cloud instance product string.
Version string of the installed system firmware.
Release date of the installed system firmware.
Manufacturer of the motherboard.
Model or part name of the motherboard.
Physical chassis form factor of the host.
Input-output memory management unit state, which gates kernel-bypass zero-copy behavior and per-packet translation cost.
Identity of the virtualization platform when virtualized, or none for bare metal.
Whether the validator runs inside a container and which runtime, useful for distinguishing bare-metal from containerized deployments.
Whether the validator is running inside a managed orchestration pod.
Hard memory limit imposed by the control group, 0 when unlimited; the ceiling the kernel kills at.
Memory throttle threshold imposed by the control group, 0 when unset; exceeding it forces reclaim stalls long before the hard limit kills.
Swap limit imposed by the control group, 0 when swap is denied to the process even though the host has swap, UInt64 max when unlimited or no limit file exists.
Processor quota imposed by the control group in millicores, 0 when unlimited; a quota below the tile count strangles the validator.
Inventory of storage devices, each with its identity, capacity, media type, transport, and configured input/output scheduler.
Per non-volatile express device information.
Per-array software storage summary.
Filesystems the validator's directories live on, one entry per distinct mount; the directory index fields below point into this array.
Index into filesystems of the mount holding the accounts database; the database grows until write failures stall replay. 255 when unset or unresolved.
Index into filesystems of the mount holding the snapshots directory; download plus unpack needs roughly twice the snapshot size, and running out mid-download is a leading boot failure. 255 when unset or unresolved.
Index into filesystems of the mount holding the log file, usually the root partition rather than the data disks. 255 when unset or unresolved.
Index into filesystems of the mount holding the shred database. 255 when unset or unresolved.
Index into filesystems of the mount holding the GUI database. 255 when unset or unresolved.
The full resolved validator configuration serialized as text, capturing every effective setting after defaults, overrides, and system-derived values are merged. Filesystem paths, the user and host names, bind, listen, and advertised addresses, external service URLs, and gossip entrypoints are redacted.
The operator's own configuration file parsed and serialized as text: only the settings they explicitly override, before defaults are merged. Values whose keys suggest paths, addresses, hosts, users, servers, or secrets are redacted.
The full runtime topology serialized as text, capturing the complete wiring of the validator: a tiles list with each tile's kind, core assignment or floating flag, consumed and produced link lists, event link, and memory-object count; a links list, referenced from the tile link lists by position, with each link's kind, depth, message size and burst; and a workspaces list with each workspace's locality domain, page size, page count and footprint. This is the authoritative wiring fingerprint (memory-object assignments are summarized as a count); the scalar fields are derived summaries. Tile locality follows from the core assignment joined against numa_cpu_to_node. Newer builds add per tile core_shared (another concurrently running pinned tile is on the same physical core) and sibling_cpu (-1 when none).
Total number of tiles in the running topology.
Total memory the validator locks at boot: workspaces, tile stacks, and private pages.
Gigantic pages the topology requires across all memory locality domains.
Huge pages the topology reserves in the hugetlbfs across all memory locality domains. Tile stack pages are reserved on every domain, since stack placement is only decided when a tile starts, so they are counted once per domain.
Normal pages the topology requires.
Wall-clock time at which the validator process itself was started.
A numeric identifier summarizing the exact set of protocol features this build understands, used to detect when nodes disagree on consensus rules.
For each logical processor, the last-level cache domain it shares (one per AMD CCD or CCX, usually one per socket on Intel), numbered from 0 in first-seen order; tiles that exchange data cheaply sit in the same domain. 65535 for an offline processor. Empty when the kernel does not expose level-3 cache topology.
Instruction set extensions this processor supports, named as in build_features; one present here but missing from build_features means the binary was built for an older processor and runs slower code paths.
For each logical processor, the physical core it runs on (keyed by package and core id), numbered from 0 in first-seen order; processors sharing a value are hyperthread siblings, so tiles on them compete for one core. 65535 for an offline processor or an unreadable core id.
Logical processors in the affinity the validator inherited from its launcher (taskset, systemd CPUAffinity, or a cgroup cpuset). Tiles without a core assignment run inside it; auto layout itself ignores it. Empty when it could not be read.
Type of the validator's cpuset partition (the cgroup named after the instance that the cpuset configure stage creates), which isolates pinned tiles from other tasks at run time.
Logical processors effectively assigned to the validator's cpuset partition. Empty when there is none.
One block storage device.
Used in:
Kernel name of the device.
Model string of the device.
Capacity of the device in bytes.
Whether the device is a spinning disk rather than solid-state.
Transport type such as the non-volatile express bus, serial bus, attached storage, or virtual.
Active input/output scheduler for the device.
Bonding mode of the validator's bonded device, if bonded; kernel-bypass support and hashing behavior differ by mode.
Used in:
Not bonded.
Failover bonding.
Link aggregation (802.3ad).
Hash-balanced bonding.
Another mode.
The named hardware target the binary was compiled for, which fixes the minimum processor it can run on. Distinct from the live processor it is currently running on.
Used in:
native
linux_gcc_x86_64
linux_gcc_zen2
linux_gcc_zen3
linux_gcc_zen4
linux_gcc_zen5
linux_gcc_icelake
linux_clang_x86_64
linux_clang_haswell
linux_clang_icelake
linux_clang_zen2
linux_clang_zen3
linux_clang_zen4
linux_clang_zen5
other
Which compiler family produced the binary.
Used in:
Built with gcc.
Built with clang.
Whether the validator runs inside a container and which runtime, useful for distinguishing bare-metal from containerized deployments.
Used in:
none
docker
podman
containerd
cri-o
lxc
systemd-nspawn
Unlisted or undeterminable.
Processor architecture of the host.
Used in:
x86-64
ARM
Unlisted or undeterminable.
Energy-performance preference in force on the frequency driver; a balance profile under governor performance silently caps sustained clocks.
Used in:
Maximum performance bias.
Default bias favoring performance.
Bias favoring power saving.
Maximum power saving.
Driver default.
Not exposed by the driver.
The kernel driver managing processor idle low-power states.
Used in:
intel_idle
acpi_idle
none
unknown
The maker of the processor.
Used in:
Intel
AMD
Unlisted or undeterminable.
The kernel driver controlling processor frequency scaling.
Used in:
intel_pstate
intel_cpufreq
acpi-cpufreq
amd-pstate
amd-pstate-epp
cppc_cpufreq
unknown
The active frequency-scaling policy, which determines how aggressively the chip clocks up.
Used in:
performance
powersave
ondemand
conservative
schedutil
userspace
unknown
Type of the validator's cpuset partition (the cgroup named after the instance that the cpuset configure stage creates), which isolates pinned tiles from other tasks at run time.
Used in:
No such cgroup, or no cpuset controller.
The cgroup exists but is not a partition, so it does not isolate.
A partition with its own scheduler domain: other tasks are kept off its processors.
A partition without scheduler domains or load balancing: other tasks are kept off its processors.
The kernel rejected the partition, so it does not isolate.
Unlisted or unreadable.
Physical chassis form factor of the host.
Used in:
Unlisted or undeterminable.
desktop
laptop
notebook
tower
rack_mount
blade
blade_enclosure
mini_pc
stick_pc
embedded
sealed_case_pc
convertible
detachable
virtual
cloud
One populated memory module.
Used in:
Module capacity in bytes.
Memory technology, such as DDR4 or DDR5.
Configured operating speed in mega-transfers per second.
Maximum rated speed in mega-transfers per second.
Module manufacturer name.
Module part number: the exact vendor SKU including speed grade and revision.
One mounted filesystem.
Used in:
Index into block_devices of the backing device; 255 when not a local block device.
Filesystem type, such as ext4, xfs, or tmpfs.
Mount options in effect. Option values that can carry paths, addresses, or usernames are redacted.
Total capacity in bytes.
Free space in bytes at boot.
Identity of the virtualization platform when virtualized, or none for bare metal.
Used in:
none
kvm
qemu
vmware
microsoft_hyperv
xen
amazon_nitro
kvm_gce
bhyve
parallels
virtualbox
bochs
unknown
Input-output memory management unit state, which gates kernel-bypass zero-copy behavior and per-packet translation cost.
Used in:
No IOMMU active.
Active in passthrough; no per-packet translation.
Active with translation; adds per-packet cost.
Not determinable.
Which system C library the validator is running against.
Used in:
glibc
musl
unknown
One software storage array.
Used in:
Array name.
Redundancy level of the array.
Number of devices in the array.
Whether the array is degraded.
One physical network interface and its hardware and link properties.
Used in:
Interface name.
Kernel device driver module backing the interface.
Version string reported by the interface driver.
Firmware version running on the interface hardware.
Negotiated PCIe link width; less than the maximum means the card trained down and lost bandwidth.
Maximum PCIe link width the device supports.
Negotiated PCIe link speed, such as 16.0 GT/s.
Maximum PCIe link speed the device supports.
Hardware bus location of the card, typically its peripheral bus address.
Peripheral bus device identifier of the card.
Peripheral bus vendor identifier of the card.
Peripheral bus subsystem vendor identifier of the card.
Peripheral bus subsystem device identifier of the card.
Memory locality domain the card is attached to; all-ones when not applicable.
Largest packet size the interface will send without fragmentation.
Negotiated link speed in megabits per second.
Whether the interface currently detects a physical link.
Whether the interface accepts all packets on the wire rather than only those addressed to it.
Hardware queue channels currently configured.
Maximum hardware queue channels the device supports.
Number of descriptor slots in the hardware receive ring buffer.
Number of descriptor slots in the hardware transmit ring buffer.
Maximum receive ring buffer size the hardware supports.
Maximum transmit ring buffer size the hardware supports.
Kernel network device features currently enabled on the interface, under their kernel names, such as rx-checksum or rx-gro; rx-lro appearing here is harmful to latency-sensitive traffic.
Whether receive-side hashing over UDP ports was successfully enabled, required to spread flows across queues.
Receive interrupt coalescing delay in microseconds.
Whether receive interrupt coalescing is tuned dynamically based on load.
Hashing function the card uses to distribute incoming packets across queues.
Number of active hardware flow-steering rules installed on the card.
Whether the interface is a bonded aggregate of multiple physical members.
One non-volatile express storage device.
Used in:
Index of this drive in block_devices, which carries its model and identity.
Firmware revision of the device.
Drive critical warning flags from the health log; any nonzero bit means the drive is degraded.
Remaining spare capacity percentage; below threshold predicts imminent read-only fallback.
Endurance used percentage; values near or past 100 mark worn media.
Media and data integrity error count; nonzero flags drives silently retrying reads.
Memory locality domain of the drive. 255 when not applicable.
Size of the device namespace in bytes.
Whether the health log was read. When false the warning, spare, used and error fields are unknown, not healthy.
Combined event type
Used in:
Metrics of one accdb tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Account versions in the index; every unrooted fork version counts
Account versions the index can hold
Accounts moved in the account database during compaction
Allocated size of the account database file
gauge: current value, do not sum over time. Bytes of the account database file written at least once, excluding never-written partition tail space
Bytes of the account database file holding live data
Whether a compaction is currently in progress
Compactions requested on the account database
Compactions fully completed on the account database, less than or equal to the requested amount
Bytes moved in the account database during compaction
Dirty accounts written back and evicted by the background preevict pass, by size class; clean lines reclaimed are not counted
Bytes read from the account database file by compaction and writeback
Bytes written to the account database file by compaction and writeback
Write syscalls that wrote evicted cache lines back to the account database file
Writes that relocated a batch of records during compaction
Account versions removed from the account database index
Cache lines in use, by size class
Cache line capacity, by size class
gauge: current value, do not sum over time. Cache lines reserved by in-flight acquires, by size class; 0 when reservation tracking is disabled for the class
Whether cache line reservations are tracked for each size class; untracked classes always have room for every joiner and read 0 in CacheClassReserved
Free cache lines the background preevict pass tries to restore, by size class
Free cache line count below which the background preevict pass starts, by size class
Bytes reclaimed from the program cache by the background preevict pass
Program cache slots reclaimed by the background preevict pass, by size class
Dirty accounts written back and evicted by the background preevict pass, by size class; clean lines reclaimed are not counted
Used in:
0-128 B
129-512 B
513 B-2 KiB
2-8 KiB
8-32 KiB
32-128 KiB
128 KiB-1 MiB
1-10 MiB
Cache line capacity, by size class
Used in:
0-128 B
129-512 B
513 B-2 KiB
2-8 KiB
8-32 KiB
32-128 KiB
128 KiB-1 MiB
1-10 MiB
Free cache line count below which the background preevict pass starts, by size class
Used in:
0-128 B
129-512 B
513 B-2 KiB
2-8 KiB
8-32 KiB
32-128 KiB
128 KiB-1 MiB
1-10 MiB
Free cache lines the background preevict pass tries to restore, by size class
Used in:
0-128 B
129-512 B
513 B-2 KiB
2-8 KiB
8-32 KiB
32-128 KiB
128 KiB-1 MiB
1-10 MiB
gauge: current value, do not sum over time. Cache lines reserved by in-flight acquires, by size class; 0 when reservation tracking is disabled for the class
Used in:
0-128 B
129-512 B
513 B-2 KiB
2-8 KiB
8-32 KiB
32-128 KiB
128 KiB-1 MiB
1-10 MiB
Cache lines in use, by size class
Used in:
0-128 B
129-512 B
513 B-2 KiB
2-8 KiB
8-32 KiB
32-128 KiB
128 KiB-1 MiB
1-10 MiB
Whether cache line reservations are tracked for each size class; untracked classes always have room for every joiner and read 0 in CacheClassReserved
Used in:
0-128 B
129-512 B
513 B-2 KiB
2-8 KiB
8-32 KiB
32-128 KiB
128 KiB-1 MiB
1-10 MiB
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Program cache slots reclaimed by the background preevict pass, by size class
Used in:
0-128 KiB
128-512 KiB
512 KiB-1 MiB
1-2 MiB
2-4 MiB
4-11 MiB
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Metrics of one admin tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Metrics of one bundle tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Transactions received, including transactions within bundles
Transactions received outside of a bundle
Bytes from received gRPC protobuf payloads
Bundles received
Bundle client failures, by cause
Workspace heap capacity
Approximate free space in workspace
HTTP/2 PINGs acknowledged by server
Block engine connection state; Sleeping means closed on purpose because the next leader slot is far away: 0=disconnected, 1=connecting, 2=connected, 3=sleeping
Latest block engine HTTP/2 PING round trip, 0 if none since connect
Smoothed block engine HTTP/2 PING round trip (RFC 9002), 0 if none since connect
Mean deviation of block engine HTTP/2 PING round trips (RFC 9002 rttvar), 0 if none since connect
Local wall clock minus server timestamp per block engine message; includes clock skew, negative clamped to 0
Transactions buffered and waiting to be published
Transactions from the block engine dropped because the verify tiles were too slow to keep up
Bundle client failures, by cause
Used in:
Protobuf decode or encode failed
gRPC transport failed
Bundle dropped due to missing fee info
Block engine connection state; Sleeping means closed on purpose because the next leader slot is far away: 0=disconnected, 1=connecting, 2=connected, 3=sleeping
Used in:
Disconnected
Connecting
Connected
Sleeping
Local wall clock minus server timestamp per block engine message; includes clock skew, negative clamped to 0
Used in:
[0, 100000) nanos
[100000, 193070) nanos
[193070, 372760) nanos
[372760, 719687) nanos
[719687, 1389498) nanos
[1389498, 2682700) nanos
[2682700, 5179482) nanos
[5179482, 10000012) nanos
[10000012, 19306997) nanos
[19306997, 37275969) nanos
[37275969, 71968616) nanos
[71968616, 138949620) nanos
[138949620, 268269670) nanos
[268269670, 517947555) nanos
[517947555, 1000000000) nanos
[1000000000, inf) nanos
Sum of samples (nanos)
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Metrics of one dedup tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Transactions processed, by result
Gossip vote transactions from pohh (Frankendancer only)
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Transactions processed, by result
Used in:
Transaction was not a duplicate
Another transaction in the bundle failed
Transaction was a duplicate
Metrics of one diag tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Bundle subsystem state: 0=disconnected (All bundle tiles disconnected), 1=connecting (At least one bundle tile connecting, none connected or sleeping), 2=connected (At least one bundle tile connected), 3=sleeping (At least one bundle tile sleeping, none connected), 4=disabled (No bundle tiles configured)
Vote subsystem state: 0=disabled (Non-voting or no votor tile), 1=not_started (Votor tile not running or no votes cast yet), 2=delinquent (Vote distance exceeds threshold or vote stalled), 3=voting (Voting normally), 4=inadmissible (Voting, but the vote account fails the validator admission ticket filter)
Replay subsystem state: 0=disabled (No replay tile), 1=not_started (Replay tile not running or slots are zero), 2=behind (Replay lagging behind turbine or reset slot stalled), 3=running (Replay keeping up)
Turbine subsystem state: 0=disabled (No shred or replay tiles), 1=not_started (Tiles not all running or turbine slot is zero), 2=stalled (Turbine slot not advancing), 3=repair_outpacing (Repair byte throughput exceeds turbine), 4=running (Turbine receiving normally)
External block builder subsystem state: 0=disabled (No block builder configured), 1=disconnected (Block builder disconnected), 2=connecting (Block builder connection in progress), 3=unhealthy (Connected, but the block builder is not in a usable state), 4=connected (Block builder connected and healthy)
counter: cumulative since tile start, query deltas. Device IRQs across all CPUs
counter: cumulative since tile start, query deltas. Device IRQs on CPUs dedicated to pinned tiles; subset of SystemDeviceIrq
counter: cumulative since tile start, query deltas. Soft-IRQs across all CPUs, by softirq type
counter: cumulative since tile start, query deltas. Soft-IRQs on CPUs dedicated to pinned tiles, by softirq type; subset of SystemSoftirq
Hottest CPU package temperature (k10temp Tctl or coretemp Package), 0 if no sensor
Hottest NIC ASIC temperature (mlx5 or ice), 0 if no sensor
Hottest NVMe composite temperature, 0 if no sensor or the kernel lacks CONFIG_NVME_HWMON
CPU package power from the powercap (RAPL) energy counters, averaged over 5 s, 0 if unavailable
Whole-system power draw from the ACPI power meter, averaged over its interval, 0 if no meter
counter: cumulative since tile start, query deltas. Corrected memory ECC errors across all memory controllers since boot (EDAC), 0 if firmware handles errors instead
counter: cumulative since tile start, query deltas. Uncorrected memory ECC errors across all memory controllers since boot (EDAC), 0 if firmware handles errors instead
Load average over each window, times 100 (/proc/loadavg has two decimals)
Runnable threads on the host (/proc/loadavg)
CPU time summed over all CPUs since the diag tile started, by state (/proc/stat), 10 ms resolution
Host memory available without swapping (/proc/meminfo available)
Host memory not in use at all (/proc/meminfo free)
counter: cumulative since tile start, query deltas. Bytes swapped in from disk since boot (pswpin)
counter: cumulative since tile start, query deltas. Bytes swapped out to disk since boot (pswpout)
counter: cumulative since tile start, query deltas. Major page faults across the host since boot (pgmajfault)
Processes killed by the kernel OOM killer since boot (oom_kill)
Time at least one task stalled waiting for CPU since boot (PSI some), 0 if the kernel lacks PSI
Time at least one task stalled on memory reclaim or swap since boot (PSI some), 0 if the kernel lacks PSI
Time all non-idle tasks stalled on memory together since boot (PSI full), a subset of MemoryPressureDurationNanos
Time at least one task stalled on IO since boot (PSI some), 0 if the kernel lacks PSI
Time all non-idle tasks stalled on IO together since boot (PSI full), a subset of IoPressureDurationNanos
Bytes read from the accounts disk since boot (/sys/block stat sectors read), 0 if not on a block device
Bytes written to the accounts disk since boot (/sys/block stat sectors written), 0 if not on a block device
Write requests the accounts disk completed since boot (/sys/block stat writes completed), 0 if not on a block device
Time write requests to the accounts disk spent from submission to completion since the diag tile started, summed over requests (/sys/block stat write ticks), 0 if not on a block device
Time the accounts disk had at least one request in flight since the diag tile started (/sys/block stat io_ticks), 0 if not on a block device
Integral of requests in flight at the accounts disk over time since the diag tile started (request-nanoseconds, /sys/block stat time_in_queue); divide by elapsed time for the average queue depth, 0 if not on a block device
Requests the accounts disk had in flight at the last sample (/sys/block stat in_flight), 0 if not on a block device
External block builder subsystem state: 0=disabled (No block builder configured), 1=disconnected (Block builder disconnected), 2=connecting (Block builder connection in progress), 3=unhealthy (Connected, but the block builder is not in a usable state), 4=connected (Block builder connected and healthy)
Used in:
No block builder configured
Block builder disconnected
Block builder connection in progress
Connected, but the block builder is not in a usable state
Block builder connected and healthy
Bundle subsystem state: 0=disconnected (All bundle tiles disconnected), 1=connecting (At least one bundle tile connecting, none connected or sleeping), 2=connected (At least one bundle tile connected), 3=sleeping (At least one bundle tile sleeping, none connected), 4=disabled (No bundle tiles configured)
Used in:
All bundle tiles disconnected
At least one bundle tile connecting, none connected or sleeping
At least one bundle tile connected
At least one bundle tile sleeping, none connected
No bundle tiles configured
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Replay subsystem state: 0=disabled (No replay tile), 1=not_started (Replay tile not running or slots are zero), 2=behind (Replay lagging behind turbine or reset slot stalled), 3=running (Replay keeping up)
Used in:
No replay tile
Replay tile not running or slots are zero
Replay lagging behind turbine or reset slot stalled
Replay keeping up
counter: cumulative since tile start, query deltas. Soft-IRQs on CPUs dedicated to pinned tiles, by softirq type; subset of SystemSoftirq
Used in:
NET_TX, NET_RX
BLOCK
e.g. TIMER, HRTIMER, IRQ_POLL, TASKLET, SCHED, RCU, ...
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
CPU time summed over all CPUs since the diag tile started, by state (/proc/stat), 10 ms resolution
Used in:
User (including nice)
System
Idle
Idle waiting on IO
Hardirq and softirq handlers
Stolen by the hypervisor
Load average over each window, times 100 (/proc/loadavg has two decimals)
Used in:
1 minute
5 minutes
15 minutes
counter: cumulative since tile start, query deltas. Soft-IRQs across all CPUs, by softirq type
Used in:
NET_TX, NET_RX
BLOCK
e.g. TIMER, HRTIMER, IRQ_POLL, TASKLET, SCHED, RCU, ...
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Turbine subsystem state: 0=disabled (No shred or replay tiles), 1=not_started (Tiles not all running or turbine slot is zero), 2=stalled (Turbine slot not advancing), 3=repair_outpacing (Repair byte throughput exceeds turbine), 4=running (Turbine receiving normally)
Used in:
No shred or replay tiles
Tiles not all running or turbine slot is zero
Turbine slot not advancing
Repair byte throughput exceeds turbine
Turbine receiving normally
Vote subsystem state: 0=disabled (Non-voting or no votor tile), 1=not_started (Votor tile not running or no votes cast yet), 2=delinquent (Vote distance exceeds threshold or vote stalled), 3=voting (Voting normally), 4=inadmissible (Voting, but the vote account fails the validator admission ticket filter)
Used in:
Non-voting or no votor tile
Votor tile not running or no votes cast yet
Vote distance exceeds threshold or vote stalled
Voting normally
Voting, but the vote account fails the validator admission ticket filter
Metrics of one event tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Telemetry server connection state: 0=disconnected, 1=connecting, 2=authenticating, 3=connected
Events in the queue, sent-but-unacknowledged plus unsent
Events in the event queue not yet sent to the event service
Oldest events overwritten because the queue was full
Bytes used in the event queue
Event queue capacity
Events written to the stream, including resends after reconnect
Acknowledgements received; one ack may retire many events
Nonce of the most recently acknowledged event, 0 if none
Bytes sent to the event service
Bytes received from the event service
Authentication failures with the event service
Malformed responses received from the event service
Connection attempts to the event service
Authentication handshakes that timed out
Reconnects forced because the event service stopped granting HTTP/2 flow-control credit while data was pending
Telemetry server connection state: 0=disconnected, 1=connecting, 2=authenticating, 3=connected
Used in:
Disconnected
Connecting
Authenticating
Connected
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Metrics of one execle tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Transactions loaded and executed, by result
Transactions processed, by whether and how they landed in the block
Transactions processed, by transaction version
Top-level instructions executed
Cross-program invocations executed
Committed writable accounts left byte-identical by their transaction, so their in-band lthash update was skipped
Compute units consumed by executed transactions
Time per transaction, by phase; a phase counts only once the next phase starts, so dropped, fees-only and no-op transactions lose their trailing phases
Time in top-level BPF invocations, by phase, with CPI setup and commit split out; nested programs, including native CPIs, count as Interpreter; subset of TxnRegime Execute
Program cache lookups; equals Hit + Miss + lookups of invalid program accounts
Program cache misses, each followed by an insert or a spill
Program cache lookups that waited on another tile's in-flight load of the same program
Program cache inserts that found no free slot in their size class on the first try and retried after eviction
Bytes inserted into the program cache
Bytes loaded into program cache spill memory
Bytes evicted from the program cache by insert eviction sweeps
Time in program cache lookups, including waits on peer loads and insert, eviction and ELF load on a miss; superset of ProgcacheLoadDurationNanos
Program ELF loads and verifications for the program cache, including failed verifications and spill-backed loads
Time spent loading programs
Program cache hits, by size class
Programs inserted into the program cache, by size class
Program cache slots reclaimed by an insert's eviction sweep, by size class (housekeeping preevicts are counted by the accdb tile)
Programs loaded into spill memory because no cache slot could be freed, by size class
Readonly accounts acquired from the account database, by cache size class of the account's current data size
Writable accounts acquired from the account database, by cache size class of the account's current data size
Accounts evicted from the account database cache to make space, by cache size class
New account versions committed to the index, by destination cache size class
Account versions committed over an existing version of the same fork and generation, by destination cache size class
Accounts not found in the account database cache and read from disk, by cache size class
Accounts that had to wait for another reader to finish loading them into the cache
Batch acquire calls made by this tile
Spin retries while waiting to reserve cache lines for a transaction
Bytes read from the account database file
Reads from the account database file
Bytes written to the account database file
Writes to the account database file
Account data bytes copied in memory between cache lines, staging buffers and callers; disk reads excluded
Cache invalidations due to transaction execution from a different block
Transaction cache queries by this tile that found a page not in memory
Transaction cache inserts by this tile that found a page not in memory
Transaction cache queries and inserts by this tile that fell back to the exclusive lock, stalling every tile; pages they load are read while holding it
Duration of one transaction cache query
Duration of one transaction cache insert
Accounts not found in the account database cache and read from disk, by cache size class
Used in:
0-128 B
129-512 B
513 B-2 KiB
2-8 KiB
8-32 KiB
32-128 KiB
128 KiB-1 MiB
1-10 MiB
Accounts evicted from the account database cache to make space, by cache size class
Used in:
0-128 B
129-512 B
513 B-2 KiB
2-8 KiB
8-32 KiB
32-128 KiB
128 KiB-1 MiB
1-10 MiB
New account versions committed to the index, by destination cache size class
Used in:
0-128 B
129-512 B
513 B-2 KiB
2-8 KiB
8-32 KiB
32-128 KiB
128 KiB-1 MiB
1-10 MiB
Account versions committed over an existing version of the same fork and generation, by destination cache size class
Used in:
0-128 B
129-512 B
513 B-2 KiB
2-8 KiB
8-32 KiB
32-128 KiB
128 KiB-1 MiB
1-10 MiB
Readonly accounts acquired from the account database, by cache size class of the account's current data size
Used in:
0-128 B
129-512 B
513 B-2 KiB
2-8 KiB
8-32 KiB
32-128 KiB
128 KiB-1 MiB
1-10 MiB
Writable accounts acquired from the account database, by cache size class of the account's current data size
Used in:
0-128 B
129-512 B
513 B-2 KiB
2-8 KiB
8-32 KiB
32-128 KiB
128 KiB-1 MiB
1-10 MiB
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Program cache slots reclaimed by an insert's eviction sweep, by size class (housekeeping preevicts are counted by the accdb tile)
Used in:
0-128 KiB
128-512 KiB
512 KiB-1 MiB
1-2 MiB
2-4 MiB
4-11 MiB
Program cache hits, by size class
Used in:
0-128 KiB
128-512 KiB
512 KiB-1 MiB
1-2 MiB
2-4 MiB
4-11 MiB
Programs inserted into the program cache, by size class
Used in:
0-128 KiB
128-512 KiB
512 KiB-1 MiB
1-2 MiB
2-4 MiB
4-11 MiB
Programs loaded into spill memory because no cache slot could be freed, by size class
Used in:
0-128 KiB
128-512 KiB
512 KiB-1 MiB
1-2 MiB
2-4 MiB
4-11 MiB
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Transactions loaded and executed, by result
Used in:
Transaction executed successfully
An error occurred while processing an instruction
The transaction fee payer address was not found
A program account referenced by the transaction was not found
The transaction fee payer did not have balance to pay the fee
The transaction fee payer account is not owned by the system program, or has data that is not a nonce
The transaction has already been processed in a recent block
The transaction references a blockhash that is not recent, or advances a nonce with the wrong value
A program account referenced by the transaction was no executable.
The transaction references an ALUT account that does not exist or is inactive
The transaction references an ALUT account that is not owned by the ALUT program account
The transaction references an ALUT account that contains data which is not a valid ALUT
The transaction references an account offset from the ALUT which does not exist
The total account data size of the loaded accounts exceeds the consensus limit
A compute budget program instruction was invoked more than once
The compute budget program was invoked and set the loaded accounts data size to zero
The transaction references a nonce account that is already advanced
The transaction is a nonce transaction but the advance instruction was not valid or failed
The transaction is a nonce transaction but the blockhash is not the correct one
The transaction is a message version that is not supported by the current feature set
The process for sanitizing the transaction failed
The transaction conflicts with another transaction in the microblock.
The transaction references the same account twice
The transaction had an invalid signature
The transaction references too many accounts.
The transaction would leave an account with a lower balance than the rent-exempt minimum
The total referenced account lamports before and after the transaction was unbalanced
The transaction was part of a bundle and an earlier transaction in the bundle failed
Transactions processed, by whether and how they landed in the block
Used in:
Transaction landed and executed successfully
Transaction landed, but was fees only and did not execute
Transaction landed, but failed to execute
Transaction did not land
Transactions processed, by transaction version
Used in:
Legacy transaction format
Version 0 transaction format
Version 1 transaction format
Time per transaction, by phase; a phase counts only once the next phase starts, so dropped, fees-only and no-op transactions lose their trailing phases
Used in:
Transaction checks and account loading
Transaction execution, including VM setup
Transaction result commit
Duration of one transaction cache insert
Used in:
[0, 10) nanos
[10, 32) nanos
[32, 101) nanos
[101, 319) nanos
[319, 1008) nanos
[1008, 3185) nanos
[3185, 10064) nanos
[10064, 31800) nanos
[31800, 100480) nanos
[100480, 317492) nanos
[317492, 1003197) nanos
[1003197, 3169857) nanos
[3169857, 10015972) nanos
[10015972, 31648020) nanos
[31648020, 100000000) nanos
[100000000, inf) nanos
Sum of samples (nanos)
Duration of one transaction cache query
Used in:
[0, 10) nanos
[10, 32) nanos
[32, 101) nanos
[101, 319) nanos
[319, 1008) nanos
[1008, 3185) nanos
[3185, 10064) nanos
[10064, 31800) nanos
[31800, 100480) nanos
[100480, 317492) nanos
[317492, 1003197) nanos
[1003197, 3169857) nanos
[3169857, 10015972) nanos
[10015972, 31648020) nanos
[31648020, 100000000) nanos
[100000000, inf) nanos
Sum of samples (nanos)
Time in top-level BPF invocations, by phase, with CPI setup and commit split out; nested programs, including native CPIs, count as Interpreter; subset of TxnRegime Execute
Used in:
VM setup
VM commit
VM setup (CPI)
VM commit (CPI)
VM interpreter execution
Metrics of one execrp tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Whether the tile is processing a replay task
Transactions loaded and executed, by result
Transactions processed, by transaction version
Ed25519 signatures checked, including failures
PoH SHA-256 iterations computed
Top-level instructions executed
Cross-program invocations executed
Committed writable accounts left byte-identical by their transaction, so their in-band lthash update was skipped
Time per transaction, by phase; a phase counts only once the next phase starts, so dropped, fees-only and no-op transactions lose their trailing phases
Time in top-level BPF invocations, by phase, with CPI setup and commit split out; nested programs, including native CPIs, count as Interpreter; subset of TxnRegime Execute
Compute units consumed by executed transactions
Program cache lookups; equals Hit + Miss + lookups of invalid program accounts
Program cache misses, each followed by an insert or a spill
Program cache lookups that waited on another tile's in-flight load of the same program
Program cache inserts that found no free slot in their size class on the first try and retried after eviction
Bytes inserted into the program cache
Bytes loaded into program cache spill memory
Bytes evicted from the program cache by insert eviction sweeps
Time in program cache lookups, including waits on peer loads and insert, eviction and ELF load on a miss; superset of ProgcacheLoadDurationNanos
Program ELF loads and verifications for the program cache, including failed verifications and spill-backed loads
Time spent loading programs
Program cache hits, by size class
Programs inserted into the program cache, by size class
Program cache slots reclaimed by an insert's eviction sweep, by size class (housekeeping preevicts are counted by the accdb tile)
Programs loaded into spill memory because no cache slot could be freed, by size class
Readonly accounts acquired from the account database, by cache size class of the account's current data size
Writable accounts acquired from the account database, by cache size class of the account's current data size
Accounts evicted from the account database cache to make space, by cache size class
New account versions committed to the index, by destination cache size class
Account versions committed over an existing version of the same fork and generation, by destination cache size class
Accounts not found in the account database cache and read from disk, by cache size class
Accounts that had to wait for another reader to finish loading them into the cache
Batch acquire calls made by this tile
Spin retries while waiting to reserve cache lines for a transaction
Bytes read from the account database file
Reads from the account database file
Bytes written to the account database file
Writes to the account database file
Account data bytes copied in memory between cache lines, staging buffers and callers; disk reads excluded
Cache invalidations due to transaction execution from a different block
Transaction cache queries by this tile that found a page not in memory
Transaction cache inserts by this tile that found a page not in memory
Transaction cache queries and inserts by this tile that fell back to the exclusive lock, stalling every tile; pages they load are read while holding it
Duration of one transaction cache query
Duration of one transaction cache insert
Accounts not found in the account database cache and read from disk, by cache size class
Used in:
0-128 B
129-512 B
513 B-2 KiB
2-8 KiB
8-32 KiB
32-128 KiB
128 KiB-1 MiB
1-10 MiB
Accounts evicted from the account database cache to make space, by cache size class
Used in:
0-128 B
129-512 B
513 B-2 KiB
2-8 KiB
8-32 KiB
32-128 KiB
128 KiB-1 MiB
1-10 MiB
New account versions committed to the index, by destination cache size class
Used in:
0-128 B
129-512 B
513 B-2 KiB
2-8 KiB
8-32 KiB
32-128 KiB
128 KiB-1 MiB
1-10 MiB
Account versions committed over an existing version of the same fork and generation, by destination cache size class
Used in:
0-128 B
129-512 B
513 B-2 KiB
2-8 KiB
8-32 KiB
32-128 KiB
128 KiB-1 MiB
1-10 MiB
Readonly accounts acquired from the account database, by cache size class of the account's current data size
Used in:
0-128 B
129-512 B
513 B-2 KiB
2-8 KiB
8-32 KiB
32-128 KiB
128 KiB-1 MiB
1-10 MiB
Writable accounts acquired from the account database, by cache size class of the account's current data size
Used in:
0-128 B
129-512 B
513 B-2 KiB
2-8 KiB
8-32 KiB
32-128 KiB
128 KiB-1 MiB
1-10 MiB
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Program cache slots reclaimed by an insert's eviction sweep, by size class (housekeeping preevicts are counted by the accdb tile)
Used in:
0-128 KiB
128-512 KiB
512 KiB-1 MiB
1-2 MiB
2-4 MiB
4-11 MiB
Program cache hits, by size class
Used in:
0-128 KiB
128-512 KiB
512 KiB-1 MiB
1-2 MiB
2-4 MiB
4-11 MiB
Programs inserted into the program cache, by size class
Used in:
0-128 KiB
128-512 KiB
512 KiB-1 MiB
1-2 MiB
2-4 MiB
4-11 MiB
Programs loaded into spill memory because no cache slot could be freed, by size class
Used in:
0-128 KiB
128-512 KiB
512 KiB-1 MiB
1-2 MiB
2-4 MiB
4-11 MiB
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Transactions loaded and executed, by result
Used in:
Transaction executed successfully
An error occurred while processing an instruction
The transaction fee payer address was not found
A program account referenced by the transaction was not found
The transaction fee payer did not have balance to pay the fee
The transaction fee payer account is not owned by the system program, or has data that is not a nonce
The transaction has already been processed in a recent block
The transaction references a blockhash that is not recent, or advances a nonce with the wrong value
A program account referenced by the transaction was no executable.
The transaction references an ALUT account that does not exist or is inactive
The transaction references an ALUT account that is not owned by the ALUT program account
The transaction references an ALUT account that contains data which is not a valid ALUT
The transaction references an account offset from the ALUT which does not exist
The total account data size of the loaded accounts exceeds the consensus limit
A compute budget program instruction was invoked more than once
The compute budget program was invoked and set the loaded accounts data size to zero
The transaction references a nonce account that is already advanced
The transaction is a nonce transaction but the advance instruction was not valid or failed
The transaction is a nonce transaction but the blockhash is not the correct one
The transaction is a message version that is not supported by the current feature set
The process for sanitizing the transaction failed
The transaction conflicts with another transaction in the microblock.
The transaction references the same account twice
The transaction had an invalid signature
The transaction references too many accounts.
The transaction would leave an account with a lower balance than the rent-exempt minimum
The total referenced account lamports before and after the transaction was unbalanced
The transaction was part of a bundle and an earlier transaction in the bundle failed
Transactions processed, by transaction version
Used in:
Legacy transaction format
Version 0 transaction format
Version 1 transaction format
Time per transaction, by phase; a phase counts only once the next phase starts, so dropped, fees-only and no-op transactions lose their trailing phases
Used in:
Transaction checks and account loading
Transaction execution, including VM setup
Transaction result commit
Duration of one transaction cache insert
Used in:
[0, 10) nanos
[10, 32) nanos
[32, 101) nanos
[101, 319) nanos
[319, 1008) nanos
[1008, 3185) nanos
[3185, 10064) nanos
[10064, 31800) nanos
[31800, 100480) nanos
[100480, 317492) nanos
[317492, 1003197) nanos
[1003197, 3169857) nanos
[3169857, 10015972) nanos
[10015972, 31648020) nanos
[31648020, 100000000) nanos
[100000000, inf) nanos
Sum of samples (nanos)
Duration of one transaction cache query
Used in:
[0, 10) nanos
[10, 32) nanos
[32, 101) nanos
[101, 319) nanos
[319, 1008) nanos
[1008, 3185) nanos
[3185, 10064) nanos
[10064, 31800) nanos
[31800, 100480) nanos
[100480, 317492) nanos
[317492, 1003197) nanos
[1003197, 3169857) nanos
[3169857, 10015972) nanos
[10015972, 31648020) nanos
[31648020, 100000000) nanos
[100000000, inf) nanos
Sum of samples (nanos)
Time in top-level BPF invocations, by phase, with CPI setup and commit split out; nested programs, including native CPIs, count as Interpreter; subset of TxnRegime Execute
Used in:
VM setup
VM commit
VM setup (CPI)
VM commit (CPI)
VM interpreter execution
Metrics of one failov tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Metrics of one genesi tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Metrics of one gossip tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Peers tracked for ping/pong, by state
Pongs processed, by result
Peers evicted from the ping tracker to make room for new ones
Peers added to the ping tracker
Peers removed from the ping tracker because they became staked
Tracked peers whose gossip address changed; pong invalidated and reset to unpinged
Messages dropped because the maximum number of sign requests were in flight
Capacity of the data store
Entries in the data store
Entries expired from the data store due to age
Entries evicted from the data store to make space for new entries
Capacity for storing peers in the data store
Peers in the data store, by stake class
Stake of peers in the data store
Peers evicted from the data store to make space for new peers
Capacity of the list of purged data store entries
Purged data store entries
Purged entries expired from the purged list due to age
Entries evicted from the purged list to make room
Verified CRDS values offered to the data store, by result
Gossip messages sent, by message type
Wire bytes sent in gossip messages, by message type
CRDS values sent in push messages, by value type
Wire bytes of CRDS values sent in push messages, by value type
CRDS values sent in pull response messages, by value type
Wire bytes of CRDS values sent in pull response messages, by value type
Staked peers with recent gossip activity
Staked peers tracked while waiting for supermajority
Stake from peers with recent gossip activity
gauge: current value, do not sum over time. Stake tracked while waiting for supermajority
Wait-for-supermajority progress: 0=unset (Not started), 1=loading_stakes (Loading stakes from the snapshot), 2=waiting_for_stake (Waiting for enough stake to come online), 3=publishing (Publishing our contact info to start), 4=done_or_disabled (Done, or wait-for-supermajority is disabled)
Peers removed from the ping tracker, by reason
Received entry versions not relayed because a newer version of the same entry replaced them within the relay interval
Client id currently advertised in our ContactInfo
Peers in the data store, by stake class
Used in:
Peer has stake
Peer has no stake
CRDS values sent in pull response messages, by value type
Used in:
Contact Info V1
Vote
Lowest Slot
Snapshot Hashes
Accounts Hashes
Epoch Slots
Version V1
Version V2
Node Instance
Duplicate Shred
Incremental Snapshot Hashes
Contact Info V2
Restart Last Voted Fork Slots
Restart Heaviest Fork
Wire bytes of CRDS values sent in pull response messages, by value type
Used in:
Contact Info V1
Vote
Lowest Slot
Snapshot Hashes
Accounts Hashes
Epoch Slots
Version V1
Version V2
Node Instance
Duplicate Shred
Incremental Snapshot Hashes
Contact Info V2
Restart Last Voted Fork Slots
Restart Heaviest Fork
CRDS values sent in push messages, by value type
Used in:
Contact Info V1
Vote
Lowest Slot
Snapshot Hashes
Accounts Hashes
Epoch Slots
Version V1
Version V2
Node Instance
Duplicate Shred
Incremental Snapshot Hashes
Contact Info V2
Restart Last Voted Fork Slots
Restart Heaviest Fork
Wire bytes of CRDS values sent in push messages, by value type
Used in:
Contact Info V1
Vote
Lowest Slot
Snapshot Hashes
Accounts Hashes
Epoch Slots
Version V1
Version V2
Node Instance
Duplicate Shred
Incremental Snapshot Hashes
Contact Info V2
Restart Last Voted Fork Slots
Restart Heaviest Fork
Verified CRDS values offered to the data store, by result
Used in:
Pull Response (upserted)
Push (upserted)
Pull Response (newer entry already present in table)
Pull Response (duplicate)
Push (newer entry already present in table)
Push (duplicate)
Entries in the data store
Used in:
Contact Info V1
Vote
Lowest Slot
Snapshot Hashes
Accounts Hashes
Epoch Slots
Version V1
Version V2
Node Instance
Duplicate Shred
Incremental Snapshot Hashes
Contact Info V2
Restart Last Voted Fork Slots
Restart Heaviest Fork
Gossip messages sent, by message type
Used in:
Pull Request
Pull Response
Push
Prune
Ping
Pong
Wire bytes sent in gossip messages, by message type
Used in:
Pull Request
Pull Response
Push
Prune
Ping
Pong
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Peers tracked for ping/pong, by state
Used in:
Peer is known but has not been pinged yet
Peer was pinged but has not yet responded, or responded with an invalid pong
Peer was pinged and is currently valid
Peer was pinged and is currently valid, and is being refreshed with a new ping
Peers removed from the ping tracker, by reason
Used in:
Evicted to make room for a new peer
Became staked, no longer pinged
Contact info not received for 60 s
Became our identity
Pongs processed, by result
Used in:
Pong accepted
Ignored because the peer is staked and needs no ping
Ignored because the peer has no outstanding ping
Ignored because it came from a different address than the one pinged
Ignored because the token did not match
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Wait-for-supermajority progress: 0=unset (Not started), 1=loading_stakes (Loading stakes from the snapshot), 2=waiting_for_stake (Waiting for enough stake to come online), 3=publishing (Publishing our contact info to start), 4=done_or_disabled (Done, or wait-for-supermajority is disabled)
Used in:
Not started
Loading stakes from the snapshot
Waiting for enough stake to come online
Publishing our contact info to start
Done, or wait-for-supermajority is disabled
Metrics of one gossvf tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Gossip messages received, by verification result
Wire bytes of gossip messages received, by verification result
CRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails
Wire bytes of CRDS values received, by verification result
CRDS values in push and pull-response messages, by outcome; Verified and Duplicate only for messages forwarded to gossip, Dropped counted as each check fails
Used in:
Pull Response (success)
Pull Response (duplicate)
Push (success)
Push (duplicate)
Pull Response (signature)
Pull Response (origin no contact info)
Pull Response (origin shred version)
Pull Response (inactive)
Pull Response (wallclock)
Push (signature)
Push (origin no contact info)
Push (origin shred version)
Push (inactive)
Push (wallclock)
Pull Response (deprecated value type)
Push (deprecated value type)
Wire bytes of CRDS values received, by verification result
Used in:
Pull Response (success)
Pull Response (duplicate)
Push (success)
Push (duplicate)
Pull Response (signature)
Pull Response (origin no contact info)
Pull Response (origin shred version)
Pull Response (inactive)
Pull Response (wallclock)
Push (signature)
Push (origin no contact info)
Push (origin shred version)
Push (inactive)
Push (wallclock)
Pull Response (deprecated value type)
Push (deprecated value type)
Gossip messages received, by verification result
Used in:
Pull Request (success)
Pull Response (success)
Push (success)
Prune (success)
Ping (success)
Pong (success)
Unparseable
Pull Request (not contact info)
Pull Request (loopback)
Pull Request (inactive)
Pull Request (wallclock)
Pull Request (signature)
Pull Request (shred version)
Pull Request (mask bits)
Prune (destination)
Prune (wallclock)
Prune (signature)
Push (loopback)
Push (no valid crds)
Pull Response (loopback)
Pull Response (no valid crds)
Ping (signature)
Pong (signature)
Wire bytes of gossip messages received, by verification result
Used in:
Pull Request (success)
Pull Response (success)
Push (success)
Prune (success)
Ping (success)
Pong (success)
Unparseable
Pull Request (not contact info)
Pull Request (loopback)
Pull Request (inactive)
Pull Request (wallclock)
Pull Request (signature)
Pull Request (shred version)
Pull Request (mask bits)
Prune (destination)
Prune (wallclock)
Prune (signature)
Push (loopback)
Push (no valid crds)
Pull Response (loopback)
Pull Response (no valid crds)
Ping (signature)
Pong (signature)
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Metrics of one gui tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Open HTTP connections to the GUI service, excluding connections upgraded to WebSocket
Open WebSocket connections to the GUI service
WebSocket frames sent to all connections to the GUI service
WebSocket frames received from all connections to the GUI service
Bytes sent on all connections to the GUI service
Bytes received on all connections to the GUI service
Logical bytes allocated by the GUI database backing file
Bytes occupied by records present in the store ring
Bytes of regions owned by each database ring; grows on demand
Bytes of free regions not owned by any database ring
Records present in the store ring
Record capacity of regions owned by each database ring; grows on demand
Key-value lookups (kv_get, kv_get_any, kv_iter_begin), by database
Time-series records appended, by database
Time-series scans, by database
Time-series records returned by scans, by database
Records evicted from the database, by database
Eviction passes that removed at least one record, by database
Regions claimed from the free list, by database
Regions returned to the free list, by database
Records dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database
Writes that evicted records before succeeding, by database
Bytes of regions owned by each database ring; grows on demand
Used in:
scheduler_counts
tile_timers
shred_events
txn_start
txn_end
Retired: no longer a database, always zero
slot
leader_slot
epoch
tile_stats
txn_waterfall
timeline_day
replay_txn
timeline_tile_day
fec_events
fec_event_archive
Eviction passes that removed at least one record, by database
Used in:
scheduler_counts
tile_timers
shred_events
txn_start
txn_end
Retired: no longer a database, always zero
slot
leader_slot
epoch
tile_stats
txn_waterfall
timeline_day
replay_txn
timeline_tile_day
fec_events
fec_event_archive
Writes that evicted records before succeeding, by database
Used in:
scheduler_counts
tile_timers
shred_events
txn_start
txn_end
Retired: no longer a database, always zero
slot
leader_slot
epoch
tile_stats
txn_waterfall
timeline_day
replay_txn
timeline_tile_day
fec_events
fec_event_archive
Key-value lookups (kv_get, kv_get_any, kv_iter_begin), by database
Used in:
scheduler_counts
tile_timers
shred_events
txn_start
txn_end
Retired: no longer a database, always zero
slot
leader_slot
epoch
tile_stats
txn_waterfall
timeline_day
replay_txn
timeline_tile_day
fec_events
fec_event_archive
Record capacity of regions owned by each database ring; grows on demand
Used in:
scheduler_counts
tile_timers
shred_events
txn_start
txn_end
Retired: no longer a database, always zero
slot
leader_slot
epoch
tile_stats
txn_waterfall
timeline_day
replay_txn
timeline_tile_day
fec_events
fec_event_archive
Records dropped because the database was full (MAP_FULL) or the KV index budget was exhausted (RING_FULL), by database
Used in:
scheduler_counts
tile_timers
shred_events
txn_start
txn_end
Retired: no longer a database, always zero
slot
leader_slot
epoch
tile_stats
txn_waterfall
timeline_day
replay_txn
timeline_tile_day
fec_events
fec_event_archive
Records evicted from the database, by database
Used in:
scheduler_counts
tile_timers
shred_events
txn_start
txn_end
Retired: no longer a database, always zero
slot
leader_slot
epoch
tile_stats
txn_waterfall
timeline_day
replay_txn
timeline_tile_day
fec_events
fec_event_archive
Records present in the store ring
Used in:
scheduler_counts
tile_timers
shred_events
txn_start
txn_end
Retired: no longer a database, always zero
slot
leader_slot
epoch
tile_stats
txn_waterfall
timeline_day
replay_txn
timeline_tile_day
fec_events
fec_event_archive
Regions claimed from the free list, by database
Used in:
scheduler_counts
tile_timers
shred_events
txn_start
txn_end
Retired: no longer a database, always zero
slot
leader_slot
epoch
tile_stats
txn_waterfall
timeline_day
replay_txn
timeline_tile_day
fec_events
fec_event_archive
Regions returned to the free list, by database
Used in:
scheduler_counts
tile_timers
shred_events
txn_start
txn_end
Retired: no longer a database, always zero
slot
leader_slot
epoch
tile_stats
txn_waterfall
timeline_day
replay_txn
timeline_tile_day
fec_events
fec_event_archive
Time-series records appended, by database
Used in:
scheduler_counts
tile_timers
shred_events
txn_start
txn_end
Retired: no longer a database, always zero
slot
leader_slot
epoch
tile_stats
txn_waterfall
timeline_day
replay_txn
timeline_tile_day
fec_events
fec_event_archive
Time-series records returned by scans, by database
Used in:
scheduler_counts
tile_timers
shred_events
txn_start
txn_end
Retired: no longer a database, always zero
slot
leader_slot
epoch
tile_stats
txn_waterfall
timeline_day
replay_txn
timeline_tile_day
fec_events
fec_event_archive
Time-series scans, by database
Used in:
scheduler_counts
tile_timers
shred_events
txn_start
txn_end
Retired: no longer a database, always zero
slot
leader_slot
epoch
tile_stats
txn_waterfall
timeline_day
replay_txn
timeline_tile_day
fec_events
fec_event_archive
Bytes occupied by records present in the store ring
Used in:
scheduler_counts
tile_timers
shred_events
txn_start
txn_end
Retired: no longer a database, always zero
slot
leader_slot
epoch
tile_stats
txn_waterfall
timeline_day
replay_txn
timeline_tile_day
fec_events
fec_event_archive
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Metrics of one iavf tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Packets successfully published to output links
Bytes of packets successfully published to output links
Received packets dropped, by reason
Packets dropped because the configured VF or bond member is not accepting traffic
GRE packets decapsulated and published; subset of PktRx
RX buffers not currently posted to the NIC
RX buffers currently posted to the NIC
Packets completed by the NIC for transmission
Bytes of packets completed by the NIC
Transmit jobs dropped, by reason
Transmit jobs dropped because no configured VF or bond member can send traffic
Loopback transmit jobs dropped because the socket send failed or accepted only part of a batch
GRE transmit jobs submitted to the NIC
TX buffers pending or submitted to the NIC
TX buffers available for a new transmit job
Packets received by IAVF VFs that are not intended for Firedancer
VLAN-tagged packets with the tag stripped in software and published; subset of PktRx
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Received packets dropped, by reason
Used in:
Undersized, oversized, or malformed Ethernet, IPv4 or UDP headers, or IPv4 options present
Destination IP does not match the bind address
No output link for the destination UDP port
GRE tunnel peer or headers invalid
GRE packet with no GRE tunnel configured
Transmit jobs dropped, by reason
Used in:
Invalid packet headers, or GRE packet exceeding the MTU
No matching route
Unsupported route type
Interface not available
Interface type not supported
No source IP address chosen
Next-hop neighbor unresolved
TX ring full or no TX buffer available
GRE tunnel addresses missing or outer route is itself tunnelled
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Metrics of one ipecho tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Open connections to the ipecho service
Connections to the ipecho service made and closed, by outcome
Bytes received on connections to the ipecho service
Bytes sent on connections to the ipecho service
Port check UDP packets sent to the joiner's UDP ports
Port check TCP connections to the joiner's TCP ports, by result
Port check TCP connections in flight
Connections to the ipecho service made and closed, by outcome
Used in:
Closed normally
Closed abnormally
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Port check TCP connections to the joiner's TCP ports, by result
Used in:
Port check connections connected to the joiner's port
Port check connections refused or errored
Port check connections timed out
Port check connections not attempted because the socket could not be created
Port check connections rejected because too many were in flight for a particular IP
Port check connections evicted because there were too many in flight connections
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Input link metrics of one consumer tile instance, one row per link per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Consumer tile kind
Input link name
Index of the input link on the consumer tile
Fragments consumed by the link reader
Bytes read by the link consumer
Fragments filtered and not consumed
Bytes read by the link consumer that were filtered
counter: cumulative since tile start, query deltas. Times the link was overrun while polling
counter: cumulative since tile start, query deltas. Fragments not processed because the link was overrun while polling
counter: cumulative since tile start, query deltas. Times the link was overrun while reading metadata
counter: cumulative since tile start, query deltas. Fragments not processed because the link was overrun while reading
Producer housekeeping intervals in which the producer was backpressured and this reliable consumer had the least credit across its outs; sampled, not an event count
Metrics of one metric tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Wall-clock time the validator started
Open HTTP connections to the Prometheus endpoint
Bytes sent in responses from the Prometheus endpoint
Bytes received in requests to the Prometheus endpoint
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Metrics of one mlx5 tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Packets successfully published to output links
Bytes of packets successfully published to output links
Received packets dropped, by reason
GRE packets decapsulated and published; subset of PktRx
RX buffers not currently posted to the NIC
RX buffers currently posted to the NIC
Packets completed by the NIC for transmission
Bytes of packets completed by the NIC
Transmit jobs dropped, by reason
GRE transmit jobs submitted to the NIC
TX buffers pending or submitted to the NIC
TX buffers available for a new transmit job
VLAN-tagged packets with the tag stripped in software and published; subset of PktRx
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Received packets dropped, by reason
Used in:
Undersized, oversized, or malformed Ethernet, IPv4 or UDP headers, or IPv4 options present
Destination IP does not match the bind address
No output link for the destination UDP port
GRE tunnel peer or headers invalid
GRE packet with no GRE tunnel configured
Transmit jobs dropped, by reason
Used in:
Invalid packet headers, or GRE packet exceeding the MTU
No matching route
Unsupported route type
Interface not available
Interface type not supported
No source IP address chosen
Next-hop neighbor unresolved
TX ring full or no TX buffer available
GRE tunnel addresses missing or outer route is itself tunnelled
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Metrics of one motor tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Metrics of one mwaitx tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Hardware idle-wait naps on the doorbell line
futex wakes issued to parked tiles, by cause
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
futex wakes issued to parked tiles, by cause
Used in:
A producer rang the doorbell for a parked consumer
A parked tile's deadline lapsed
The verifying sweep found a pending frag whose doorbell raced or was lost
Metrics of one net tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Packets successfully published to output links
Bytes of packets successfully published to output links
Received packets dropped, by reason
Times RX stalled because the fill ring had no room; the packet stays in the RX ring and is retried (kernel drops are XdpRxRingFullDropped)
Receive buffers currently in use
Receive buffers currently free
Packet transmit jobs submitted
Packet transmit jobs marked as completed by the kernel
Bytes of packets submitted for transmission
Transmit jobs dropped, by reason
Transmit buffers currently in use
Transmit buffers currently free
XSK sendto syscalls dispatched; in busy-poll mode each also counts in XskSyscallRx
XSK recvmsg syscalls dispatched
Dropped for other reasons (xdp_statistics_v0.rx_dropped)
Dropped due to invalid descriptor (xdp_statistics_v0.rx_invalid_descs)
Dropped due to invalid descriptor (xdp_statistics_v0.tx_invalid_descs)
Dropped due to rx ring being full (xdp_statistics_v1.rx_ring_full)
counter: cumulative since tile start, query deltas. Times the kernel found the fill ring empty (xdp_statistics_v1.rx_fill_ring_empty_descs)
counter: cumulative since tile start, query deltas. Times the kernel found the TX ring empty (xdp_statistics_v1.tx_ring_empty_descs)
GRE packets decapsulated and published; subset of PktRx
GRE packet transmit jobs submitted; subset of PktTxSubmitted
IPv4 routes installed in the forwarding table
VLAN-tagged packets with the tag stripped in software and published; subset of PktRx
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Received packets dropped, by reason
Used in:
Undersized, oversized, or malformed Ethernet, IPv4 or UDP headers, or IPv4 options present
Destination IP does not match the bind address
No output link for the destination UDP port
GRE tunnel peer or headers invalid
GRE packet with no GRE tunnel configured
Transmit jobs dropped, by reason
Used in:
Invalid packet headers, or GRE packet exceeding the MTU
No matching route
Unsupported route type
Interface not available
Interface type not supported
No source IP address chosen
Next-hop neighbor unresolved
TX ring full or no TX buffer available
GRE tunnel addresses missing or outer route is itself tunnelled
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
IPv4 routes installed in the forwarding table
Used in:
Local
Main
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Metrics of one netlnk tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Netlink receive buffer overflows (ENOBUFS); each forces a full resync
Full link table syncs
Full route table syncs
Netlink live updates processed, by message type
Network interfaces in the interface table
Neighbor solicit requests from net tiles, by result
Neighbor solicit requests from net tiles, by result
Used in:
Neighbor solicit sent to the kernel
Kernel already resolving this neighbor
Neighbor table had no room for a placeholder entry
Netlink send failed
Per-host probe rate limit hit
Global probe rate limit hit
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Netlink live updates processed, by message type
Used in:
Link
Neighbor Table Entry
IPv4 Route Table Entry
Metrics of one pack tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Duration of scheduling one microblock
Duration of discovering that no transaction is schedulable
Duration of inserting one transaction into the pending pool
Duration of marking one microblock as complete
Transactions in a scheduled microblock, including both votes and non-votes
Simple vote transactions in a scheduled microblock
Transactions received from resolv, including bundle members
Insert attempts, by result; bundle members are counted individually with the bundle's result
Time pack spent, by scheduling regime
Transactions dropped from pack after insertion, by reason
Transactions evicted from pack to make room for others, including the bundle initializer transaction removed when a leader slot ends
Pending transactions in pack's pool, by type
Maximum pending transactions pack can hold
Lower bound on the cost of the cheapest pending transaction, 0 if none pending
Times pack did not schedule a microblock because the block microblock limit was reached
Times pack did not schedule a microblock because the block byte limit was reached
Times a pending transaction was considered for a microblock, by outcome; one transaction may be considered many times
Times pack checked whether the bundle tip state needed cranking, by outcome
Cost units used in the current block, 0 if not packing a block
Cost units scheduled for each block pack produced; can exceed the block limit because of rebates
Compute units rebated for each block pack produced. Compute units are rebated when a transaction fails prior to execution or requests more compute units than it uses
Net cost units (scheduled minus rebated) in each block pack produced
Percent of max_cost_per_block used by each block pack produced
Bundles that arrived while not leader that pack is holding
Bundles that arrived while not leader that pack kept
Bundles that arrived while not leader that pack dropped because it was full of ones that declare at least as large a tip
Bundles that arrived while not leader that pack dropped for one that declares a larger tip
Bundles that arrived while not leader that pack scheduled
Batch auctions started
Batch auctions skipped because the block was already at their CU limit
Batch auctions ended, by reason
Duration that batch auctions ran for
Duration that batch auctions ran for
Used in:
[0, 10000) nanos
[10000, 19307) nanos
[19307, 37276) nanos
[37276, 71969) nanos
[71969, 138950) nanos
[138950, 268270) nanos
[268270, 517948) nanos
[517948, 1000001) nanos
[1000001, 1930699) nanos
[1930699, 3727596) nanos
[3727596, 7196860) nanos
[7196860, 13894960) nanos
[13894960, 26826964) nanos
[26826964, 51794753) nanos
[51794753, 100000000) nanos
[100000000, inf) nanos
Sum of samples (nanos)
Batch auctions ended, by reason
Used in:
An auction ended because the block reached the auction's CU limit
An auction ended because a new auction was started
An auction ended because the final 10ms of our leader rotation began
An auction ended because the leader slot ended
Times pack checked whether the bundle tip state needed cranking, by outcome
Used in:
On-chain state was already up to date
Inserted an initializer bundle to update the on-chain state
Tried to insert an initializer bundle to update the on-chain state, but creation failed
Tried to insert an initializer bundle to update the on-chain state, but insertion failed
Net cost units (scheduled minus rebated) in each block pack produced
Used in:
[0, 1000000)
[1000000, 1389495)
[1389495, 1930697)
[1930697, 2682695)
[2682695, 3727593)
[3727593, 5179474)
[5179474, 7196856)
[7196856, 9999999)
[9999999, 13894954)
[13894954, 19306976)
[19306976, 26826957)
[26826957, 37275936)
[37275936, 51794746)
[51794746, 71968567)
[71968567, 100000000)
[100000000, inf)
Sum of samples
Percent of max_cost_per_block used by each block pack produced
Used in:
[0, 1) pct
[1, 2) pct
[2, 3) pct
[3, 4) pct
[4, 5) pct
[5, 7) pct
[7, 9) pct
[9, 12) pct
[12, 16) pct
[16, 22) pct
[22, 30) pct
[30, 41) pct
[41, 55) pct
[55, 74) pct
[74, 100) pct
[100, inf) pct
Sum of samples (pct)
Cost units scheduled for each block pack produced; can exceed the block limit because of rebates
Used in:
[0, 1000000)
[1000000, 1479160)
[1479160, 2187914)
[2187914, 3236274)
[3236274, 4786966)
[4786966, 7080688)
[7080688, 10473469)
[10473469, 15491934)
[15491934, 22915046)
[22915046, 33895014)
[33895014, 50136141)
[50136141, 74159362)
[74159362, 109693545)
[109693545, 162254278)
[162254278, 240000000)
[240000000, inf)
Sum of samples
Compute units rebated for each block pack produced. Compute units are rebated when a transaction fails prior to execution or requests more compute units than it uses
Used in:
[0, 1000000)
[1000000, 1479160)
[1479160, 2187914)
[2187914, 3236274)
[3236274, 4786966)
[4786966, 7080688)
[7080688, 10473469)
[10473469, 15491934)
[15491934, 22915046)
[22915046, 33895014)
[33895014, 50136141)
[50136141, 74159362)
[74159362, 109693545)
[109693545, 162254278)
[162254278, 240000000)
[240000000, inf)
Sum of samples
Duration of marking one microblock as complete
Used in:
[0, 10) nanos
[10, 32) nanos
[32, 101) nanos
[101, 319) nanos
[319, 1008) nanos
[1008, 3185) nanos
[3185, 10064) nanos
[10064, 31800) nanos
[31800, 100480) nanos
[100480, 317492) nanos
[317492, 1003197) nanos
[1003197, 3169857) nanos
[3169857, 10015972) nanos
[10015972, 31648020) nanos
[31648020, 100000000) nanos
[100000000, inf) nanos
Sum of samples (nanos)
Duration of scheduling one microblock
Used in:
[0, 10) nanos
[10, 32) nanos
[32, 101) nanos
[101, 319) nanos
[319, 1008) nanos
[1008, 3185) nanos
[3185, 10064) nanos
[10064, 31800) nanos
[31800, 100480) nanos
[100480, 317492) nanos
[317492, 1003197) nanos
[1003197, 3169857) nanos
[3169857, 10015972) nanos
[10015972, 31648020) nanos
[31648020, 100000000) nanos
[100000000, inf) nanos
Sum of samples (nanos)
Duration of discovering that no transaction is schedulable
Used in:
[0, 10) nanos
[10, 32) nanos
[32, 101) nanos
[101, 319) nanos
[319, 1008) nanos
[1008, 3185) nanos
[3185, 10064) nanos
[10064, 31800) nanos
[31800, 100480) nanos
[100480, 317492) nanos
[317492, 1003197) nanos
[1003197, 3169857) nanos
[3169857, 10015972) nanos
[10015972, 31648020) nanos
[31648020, 100000000) nanos
[100000000, inf) nanos
Sum of samples (nanos)
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Time pack spent, by scheduling regime
Used in:
No microblock: not leader, no transactions, no available execle
No microblock: not leader, transactions available, no available execle
No microblock: not leader, no transactions, execle available
No microblock: not leader, transactions available, execle available
No microblock: leader, no transactions, no available execle
No microblock: leader, transactions available, no available execle
No microblock: leader, no transactions, execle available
No microblock: leader, transactions available, execle available (all conflicting)
Scheduled a microblock: not leader, no transactions, no available execle
Scheduled a microblock: not leader, transactions available, no available execle
Scheduled a microblock: not leader, no transactions, execle available
Scheduled a microblock: not leader, transactions available, execle available
Scheduled a microblock: leader, no transactions, no available execle
Scheduled a microblock: leader, transactions available, no available execle
Scheduled a microblock: leader, no transactions, execle available
Scheduled a microblock: leader, transactions available, execle available
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Transactions dropped from pack after insertion, by reason
Used in:
TTL expired
Already executed in the replay or leader pipeline
Part of a partial bundle
Insert attempts, by result; bundle members are counted individually with the bundle's result
Used in:
Transaction has an instruction that references too many accounts
Bundle with two conflicting durable nonce transactions
Transaction uses an account on the account blocklist
Transaction uses an account on the bundle blocklist
Transaction is an invalid durable nonce transaction
Transaction tries to write to a sysvar
Estimating compute cost or fee failed
Transaction included an account address twice
Transaction tried to load too many accounts
Transaction requests too many CUs
Transaction already expired
Transaction loads accounts from a lookup table, which is not supported here
Fee payer's balance below transaction fee
Pack aware of transaction with same signature
Transaction's fee was too low given its compute unit requirement and another competing transactions that uses the same durable nonce
Transaction's fee was too low given its compute unit requirement and other competing transactions
Transaction added to pending transactions
Simple vote transaction was added to pending transactions
Transaction replaced a lower priority transaction
Simple vote transaction replaced a lower priority transaction
Durable nonce transaction added to pending transactions
Never produced
Durable nonce transaction replaced a lower priority transaction, likely one that uses the same durable nonce
Duration of inserting one transaction into the pending pool
Used in:
[0, 10) nanos
[10, 32) nanos
[32, 101) nanos
[101, 319) nanos
[319, 1008) nanos
[1008, 3185) nanos
[3185, 10064) nanos
[10064, 31800) nanos
[31800, 100480) nanos
[100480, 317492) nanos
[317492, 1003197) nanos
[1003197, 3169857) nanos
[3169857, 10015972) nanos
[10015972, 31648020) nanos
[31648020, 100000000) nanos
[100000000, inf) nanos
Sum of samples (nanos)
Pending transactions in pack's pool, by type
Used in:
Transactions in the main treap
Deprecated, always 0 since Firedancer v26.11.
Transactions that write to a hotly-contended account
Transactions that are part of a bundle
Transactions that are currently held waiting for the next auction
Transactions in a scheduled microblock, including both votes and non-votes
Used in:
[0, 1)
[1, 2)
[2, 3)
[3, 4)
[4, 5)
[5, 6)
[6, 8)
[8, 10)
[10, 13)
[13, 17)
[17, 22)
[22, 29)
[29, 38)
[38, 49)
[49, 64)
[64, inf)
Sum of samples
Times a pending transaction was considered for a microblock, by outcome; one transaction may be considered many times
Used in:
Pack included the transaction in the microblock
Pack skipped the transaction because it would have exceeded the block CU limit
Pack skipped the transaction because of an account conflict found by the fast bitvector check
Pack skipped the transaction because it would have exceeded the block data size limit
Pack skipped the transaction because it would have exceeded the data allocation limit
Pack skipped the transaction because it would have caused a writable account to exceed the per-account block write cost limit
Pack skipped the transaction because of an account conflict found by the full check
Pack skipped the transaction because it exceeded the per-account block write cost limit too many times before
Simple vote transactions in a scheduled microblock
Used in:
[0, 1)
[1, 2)
[2, 3)
[3, 4)
[4, 5)
[5, 6)
[6, 8)
[8, 10)
[10, 13)
[13, 17)
[17, 22)
[22, 29)
[29, 38)
[38, 49)
[49, 64)
[64, inf)
Sum of samples
Metrics of one quic tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Txns evicted from reassembly before completing (txn_reassembly_count too small)
Fragmented txn receive ops started
Fragmented txn reassemblies in progress
Txn stream chunks received for reassembly
Txn stream chunks dropped because of a data gap
Txn stream chunks dropped as duplicates of a completed stream
Transactions published to verify, by TPU ingress path; excludes all *Dropped
Txn reassemblies abandoned because the connection was lost
Txns received via QUIC dropped because they were too small
Txns received via QUIC dropped because they were too large
Packets received on the non-QUIC port that were too small to be a valid IP packet
Packets received on the non-QUIC port that were too large to be a valid transaction
UDP datagrams received (one may carry several coalesced QUIC packets)
Bytes of UDP datagrams received, including Ethernet, IPv4 and UDP headers
UDP datagrams sent
Bytes of UDP datagrams sent, including Ethernet, IPv4 and UDP headers
Connection slots in use, by state; sum + ConnFree = ConnCapacity
Connection slots free
Connection slots
Connections created
Connections gracefully closed
Connections aborted
Connections timed out
Connections established with retry
Connections that failed to create due to lack of slots
Retries rejected: an invalid or expired token as server, a late or invalid Retry packet as client
Received packets dropped, by reason
Packets with an unknown connection ID, by packet type
Attempts to acquire packet metadata for transmission, by result
Initial packets received, by token type
Handshake flows created
Handshakes dropped due to alloc fail
Handshakes dropped due to eviction
Stream receive events
Stream payload bytes received
QUIC frames received, by frame type
Received packets processed for acknowledgement, by result
Duration of one fd_quic_service call that processed a due connection timer
Duration of processing one received packet
QUIC frames that failed to parse
Packets that failed decryption, by encryption level
Packets dropped because the decryption key was missing, by encryption level
QUIC Retry packets sent
QUIC packets retransmitted, by encryption level
Initials with an expired Retry token, also counted in ConnRetryRejected. Each gets an INVALID_TOKEN close so the client reconnects
Connection slots in use, by state; sum + ConnFree = ConnCapacity
Used in:
Handshaking peer
Handshake complete, confirming with peer
Active connection
Peer requested close
Connection terminating due to error
Connection is closing
Connection about to be freed
QUIC frames received, by frame type
Used in:
Unknown frame type
ACK frame
RESET_STREAM frame
STOP_SENDING frame
CRYPTO frame
NEW_TOKEN frame
STREAM frame
MAX_DATA frame
MAX_STREAM_DATA frame
MAX_STREAMS frame
DATA_BLOCKED frame
STREAM_DATA_BLOCKED frame
STREAMS_BLOCKED(bidi) frame
NEW_CONN_ID frame
RETIRE_CONN_ID frame
PATH_CHALLENGE frame
PATH_RESPONSE frame
CONN_CLOSE(transport) frame
CONN_CLOSE(app) frame
HANDSHAKE_DONE frame
PING frame
PADDING frame
DATAGRAM frame
Initial packets received, by token type
Used in:
No token
fd_quic retry token
Token of another length
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Received packets processed for acknowledgement, by result
Used in:
Packet had no packet number to acknowledge
New ACK range
Merged into existing ACK range
Out of buffers
ACK suppressed by handler
Packets that failed decryption, by encryption level
Used in:
Initial
Early data
Handshake
App data
Received packets dropped, by reason
Used in:
QUIC packet too small
QUIC packet too large
Wrong source IP
Invalid QUIC header
Invalid IPv4 or UDP header
Version negotiation or unsupported version
Packet number already received or too old
Attempts to acquire packet metadata for transmission, by result
Used in:
Success
Packet metadata pool empty
Connection reached its in-flight frame limit
Packets with an unknown connection ID, by packet type
Used in:
Initial
Retry
Handshake
1-RTT
Packets dropped because the decryption key was missing, by encryption level
Used in:
Initial
Early data
Handshake
App data
QUIC packets retransmitted, by encryption level
Used in:
Initial
Early data
Handshake
App data
Duration of processing one received packet
Used in:
[0, 10) nanos
[10, 32) nanos
[32, 101) nanos
[101, 319) nanos
[319, 1008) nanos
[1008, 3185) nanos
[3185, 10064) nanos
[10064, 31800) nanos
[31800, 100480) nanos
[100480, 317492) nanos
[317492, 1003197) nanos
[1003197, 3169857) nanos
[3169857, 10015972) nanos
[10015972, 31648020) nanos
[31648020, 100000000) nanos
[100000000, inf) nanos
Sum of samples (nanos)
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Duration of one fd_quic_service call that processed a due connection timer
Used in:
[0, 10) nanos
[10, 32) nanos
[32, 101) nanos
[101, 319) nanos
[319, 1008) nanos
[1008, 3185) nanos
[3185, 10064) nanos
[10064, 31800) nanos
[31800, 100480) nanos
[100480, 317492) nanos
[317492, 1003197) nanos
[1003197, 3169857) nanos
[3169857, 10015972) nanos
[10015972, 31648020) nanos
[31648020, 100000000) nanos
[100000000, inf) nanos
Sum of samples (nanos)
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Transactions published to verify, by TPU ingress path; excludes all *Dropped
Used in:
TPU/UDP
TPU/QUIC unfragmented
TPU/QUIC fragmented
Metrics of one repair tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Network packets sent, including requests, pings and pongs
Signed repair messages queued, by type; window types send one per missing shred (up to 32), others one per attempt
Highest slot fully delivered to replay, 0 if none
Highest slot a shred was received for, 0 if none
First turbine slot seen, the catchup target, 0 if none
Highest slot past the window with a verified cert, the staked catchup target, 0 if none
Blocks in the eager, notar and final block treaps, waiting for discovery
Pending request attempts awaiting a response; a hedge is another attempt
FEC sets published to replay
Shreds received at or below the root, ignored
Data shreds received as repair responses, by the pending request they matched
Alpenglow repair metadata responses received
Metadata responses that failed to decode
Metadata responses matching no outstanding request
ParentFecSetCount responses accepted
ParentFecSetCount responses that failed verification
FecSetRoot responses accepted
FecSetRoot responses that failed verification
Root advanced messages received from replay
Missing FEC messages received from replay, each a from-root reconsume
Pings rejected, by reason
Time from sending a repair request to receiving its response
Delay scheduled before a request's next attempt: its peer's hedge time, 16.8 ms to 100 ms, 16.8 ms after an escape
Timeouts in the eager, notar and final timeout queues
Timeouts dropped because their block is gone or their work is done
Timeouts deferred because no peer could be picked
Request attempts after the first for the same timer
Pending requests with no response after about 1 s (PENDING_TTL); a peer is banned for 10 s after 32 in a row, pending-pool evictions not counted
WindowIndexForBlockId requests that expired unanswered, at or above the root
Retries sent to a random staked peer because 3 attempts in a row brought no new shred of the FEC set
Request attempts sent after 3 or more earlier attempts for the same timer
Uncapped hedge time (srtt+4*rttvar, 100 ms if unmeasured) of each staked peer an escaped retry was sent to
Peers banned for 10 s after 4 WindowIndexForBlockId requests in a row expired, whatever their other replies
Requests sent to elicit a ping from a peer with a new address
Cluster-wide wait for turbine before a missing FEC set is repaired (srtt+4*rttvar, clamped to 20-250 ms; 250 ms until sampled)
Messages in the sign pool waiting for a signature from the sign tiles
Of TimeoutNoPeerDeferred, retries of a block not known only from turbine, which pick only staked peers: every staked peer was banned, without an address or at its in-flight limit
Timeouts put back without sending a request, by reason, one per deferral
Timeouts dropped because their role's timeout queue was full; the block goes back to its treap and the work is discovered again
Blocks whose discovery stopped because their role's timeout queue filled; the block goes back to its treap and its remaining FEC sets are queued later
Identity switches that halted signing; no request, ping or pong is signed until the switch completes and unhalts
Time the earliest due timeout has waited past its due time when metrics were written, 0 if none is due; grows while signing is halted or fewer than 40 sign slots are free
First certified slot in the window seen by a staked node, which starts its catchup, 0 if none yet
Uncapped hedge time (srtt+4*rttvar, 100 ms if unmeasured) of each staked peer an escaped retry was sent to
Used in:
[0, 10000000) nanos
[10000000, 16378937) nanos
[16378937, 26826958) nanos
[26826958, 43939706) nanos
[43939706, 71968568) nanos
[71968568, 117876865) nanos
[117876865, 193069775) nanos
[193069775, 316227769) nanos
[316227769, 517947472) nanos
[517947472, 848342904) nanos
[848342904, 1389495502) nanos
[1389495502, 2275845935) nanos
[2275845935, 3727593730) nanos
[3727593730, 6105402305) nanos
[6105402305, 10000000000) nanos
[10000000000, inf) nanos
Sum of samples (nanos)
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Pings rejected, by reason
Used in:
Failed to decode
Sender is not a known peer
Signature did not verify
Pong not sent: signing halted for an identity switch, or no free sign slot
Pong not sent: sender address differs from the peer's gossip address
Pong not sent: no request was sent to the peer within the ping TTL
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Signed repair messages queued, by type; window types send one per missing shred (up to 32), others one per attempt
Used in:
Pong
Shred at an index
Highest shred at or above an index
Ancestors of an orphan slot
Parent and FEC set count of a block (Alpenglow)
Merkle root of a FEC set (Alpenglow)
Shred at an index of a specific block (Alpenglow)
Time from sending a repair request to receiving its response
Used in:
[0, 10000000) nanos
[10000000, 13894955) nanos
[13894955, 19306977) nanos
[19306977, 26826958) nanos
[26826958, 37275937) nanos
[37275937, 51794747) nanos
[51794747, 71968568) nanos
[71968568, 100000001) nanos
[100000001, 138949551) nanos
[138949551, 193069775) nanos
[193069775, 268269582) nanos
[268269582, 372759375) nanos
[372759375, 517947471) nanos
[517947471, 719685675) nanos
[719685675, 1000000000) nanos
[1000000000, inf) nanos
Sum of samples (nanos)
Delay scheduled before a request's next attempt: its peer's hedge time, 16.8 ms to 100 ms, 16.8 ms after an escape
Used in:
[0, 10000000) nanos
[10000000, 18615210) nanos
[18615210, 34652605) nanos
[34652605, 64506553) nanos
[64506553, 120080305) nanos
[120080305, 223532013) nanos
[223532013, 416109543) nanos
[416109543, 774596665) nanos
[774596665, 1441927981) nanos
[1441927981, 2684179260) nanos
[2684179260, 4996656140) nanos
[4996656140, 9301380482) nanos
[9301380482, 17314715371) nanos
[17314715371, 32231706785) nanos
[32231706785, 60000000000) nanos
[60000000000, inf) nanos
Sum of samples (nanos)
Data shreds received as repair responses, by the pending request they matched
Used in:
Response to a pending WindowIndexForBlockId request
Response to a pending WindowIndex or HighestWindowIndex request
Matched no pending request: a duplicate, or an answer to an expired or evicted request
Repair response that showed the slot's leader equivocating, dropped before matching; only unverified-nonce HighestWindowIndex or Orphan replies
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Timeouts put back without sending a request, by reason, one per deferral
Used in:
HighestWindowIndex put back: turbine delivered a FEC set of the slot within the eager wait
FEC set request put back: every data shred arrived, the set has not completed
Metrics of one replay tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Identity account balance at the optimistically confirmed slot, or the finalized slot under Alpenglow
Our active stake at the optimistically confirmed slot, or the finalized slot under Alpenglow
Cluster active stake at the optimistically confirmed slot, or the finalized slot under Alpenglow
gauge: current value, do not sum over time. Credits our vote account earned this epoch, as of the optimistically confirmed slot, or the finalized slot under Alpenglow, where they are lamport-denominated reward credits
Whether our vote account's stake is not admitted: at the optimistically confirmed slot it fails the validator admission ticket filter (missing, below the V4 rent-exempt minimum, or not V4 with a BLS pubkey), or it passes but has not yet been admitted at an epoch boundary
Latest slot for which our vote appears in a reward certificate, 0 if none observed
Duration of ingesting one queried FEC set into the scheduler
FEC sets queried from the store
FEC set store queries that found no entry
FEC payloads spilled from the store RAM cache to disk
FEC payload bytes spilled from the store RAM cache to disk
Spilled FEC payloads read from disk
Spilled FEC payload bytes read from disk
FEC payload slots in use in the store RAM cache
FEC payload capacity of the store RAM cache
gauge: current value, do not sum over time. FEC payload views currently pinned against eviction
Whether replay has caught up to the cluster tip; set once per run, never cleared
Most recent consensus root, 0 if none
Slots between the current reset slot and our root
Slot at which we are currently leader, or 0 if none
Slot at which we are next leader, or 0 if none. If we are currently leader, this is the same as the current leader slot
Slot replay last reset to (the newest replayed slot or parent-ready parent); can decrease
Maximum number of banks we can have alive
Banks we currently have alive
Free FEC sets in the reassembly queue, 0 under Alpenglow (no reassembly queue)
Slot of the FEC set most recently dequeued from reassembly for replay, 0 if none
FEC set index of the FEC set most recently dequeued from reassembly for replay; read with LastReassemblyDequeuedSlot
Bank pool index the replay scheduler is dispatching work for, ULONG_MAX if none
Bank pool index the replay scheduler last dispatched work for, ULONG_MAX if none
Wall-clock time the replay scheduler last dispatched work, 0 if never
Staging lanes currently occupied in the replay scheduler. Up to four staging lanes can be occupied at once
High watermark of staging lanes that were at one point occupied in the replay scheduler
Bank pool index at the head of each replay staging lane, ULONG_MAX if the lane is empty
Transaction pool entries held by the replay scheduler, from parse until release
Maximum transactions the replay scheduler can hold
Microblocks available to be scheduled in the replay scheduler
Maximum microblocks the replay scheduler can hold
Blocks alive in the replay scheduler
Maximum blocks the replay scheduler can track
Blocks added to the replay scheduler, by staging state
Blocks the replay scheduler considers fully replayed
Blocks the replay scheduler abandoned for any reason
Blocks the replay scheduler found invalid while parsing or verifying PoH; also counted in SchedBlockAbandoned once drained; replay-tile verdicts not counted
Blocks the replay scheduler promoted from unstaged to a staging lane
Blocks the replay scheduler demoted from a staging lane to unstaged
Times the replay scheduler stopped dispatching a block, by reason
Times the replay scheduler activated a block on a different fork than the previously active block
Times the replay scheduler promoted a block onto a staging lane
Times the replay scheduler demoted a block from a staging lane
Forks the replay scheduler has observed
Transactions with address lookup tables the replay scheduler tried to resolve, by result
Transactions that were parsed but ended up being in abandoned blocks
Transactions that were executed but ended up being in abandoned blocks
Transactions that were fully replayed but ended up being in abandoned blocks
Integral of in-flight transactions over time (transaction-nanoseconds); divide by SchedTxnBusyDurationNanos for the average in-flight count
Time the replay scheduler had at least one transaction in flight
Time the replay scheduler had no transaction in flight
Transactions the replay pipeline has parsed
Transactions the replay pipeline has executed
Transactions the replay pipeline has sigverified
Transactions the replay pipeline has performed PoH mixin on
Transactions that were fully replayed
Microblocks the replay pipeline has parsed
Microblocks the replay pipeline has fully hashed
Microblocks that were fully replayed
Bytes the replay scheduler has ingested
Trailing bytes of a block the replay scheduler ingested but did not parse
Bytes the replay scheduler refused to ingest because the block was abandoned
FEC sets the replay scheduler has been given
Banks frozen, including our own leader blocks
gauge: current value, do not sum over time. Transactions from genesis through the last replayed block, 0 if none; drops on a fork switch
Replay loop iterations that processed no FEC set, by reason
FEC sets dropped because replay could not resolve the block's parent (Alpenglow), which was dropped on a dead lineage, evicted with its tracking slot reused, or rooted past
Times the storage root could not advance to the consensus root yet
Program cache entries rooted
Program cache value slots in use, by size class
Program cache value capacity, by size class
Readonly accounts acquired from the account database, by cache size class of the account's current data size
Writable accounts acquired from the account database, by cache size class of the account's current data size
Accounts evicted from the account database cache to make space, by cache size class
New account versions committed to the index, by destination cache size class
Account versions committed over an existing version of the same fork and generation, by destination cache size class
Accounts not found in the account database cache and read from disk, by cache size class
Accounts that had to wait for another reader to finish loading them into the cache
Batch acquire calls made by this tile
Spin retries while waiting to reserve cache lines for a transaction
Bytes read from the account database file
Reads from the account database file
Bytes written to the account database file
Writes to the account database file
Account data bytes copied in memory between cache lines, staging buffers and callers; disk reads excluded
Whether the runtime has loaded a bank and can replay
Repair FEC sets of a slot at or below the published root, or of a block whose parent is below it, skipped rather than dropped: no redelivery can replay them
Leader windows given up because the cluster already produced one of their slots, eg. our own past slots replayed after a restart (Alpenglow)
Leader windows given up after votor sent them because certificates already let a later window start (Alpenglow)
Transaction cache lookups under the read lock that found their page not resident
Dirty transaction cache pages written to the spill file on eviction
Transaction cache pages read back from the spill file
Transaction cache frames evicted to load another page
Time in transaction cache spill file reads and writes
Transaction cache logical pages in use, resident or spilled
Maximum transaction cache logical pages
Transaction cache in-memory frames holding a page
Transaction cache in-memory frames, from runtime.transaction_cache_size_mib
Time all tiles spent waiting to take the transaction cache shared lock
Time all tiles spent waiting to take the transaction cache exclusive lock
Transactions the replay scheduler moved into a staging lane when it promoted their block
Blocks that fell back to in-band lthash because every demotion reservation was in use
Writes the replay scheduler saw for out-of-band lthash, by result
Writes the replay scheduler saw for out-of-band lthash, the same as SchedLthashWriteObserved, by type
Out-of-band lthash addition pseudo transactions the dispatcher handed back
Addition pseudo transactions the dispatcher handed out as ready
Addition pseudo transactions surfaced after a later write made them stale, completed without hashing
Additions handed to an exec tile
Addition results dropped because a later write made them stale while on the tile
Addition results added to a block's delta
Applied additions subtracted back out after a later write
Speculative additions created
Speculative additions applied; a subset of SchedLthashAddApplied
Speculative additions invalidated before they applied
Lane map entries inserted, each owing one out-of-band lthash subtraction
Subtractions handed to an exec tile
Subtraction results subtracted from a block's delta
Accounts whose out-of-band lthash subtraction and addition are in the delta of a block whose end task completed
Out-of-band lthash work discarded when a block was abandoned, demoted, or switched to in-band hashing, by type
Out-of-band lthash tasks handed to exec tiles, each a batch of the accounts SchedLthashSubDispatched and SchedLthashAddDispatched count, by type
Out-of-band lthash addition batches held because a transaction or standalone speculation surfaced behind them
Dispatch polls that held an out-of-band lthash addition batch because too few exec tiles were idle
Rooted stake delegations held in memory
Rooted stake delegations that fit in memory, from runtime.stake_delegation_cache_cnt
Rooted stake delegations currently spilled to disk
Unrooted stake delegation fork deltas held in memory
Unrooted stake delegation fork deltas that fit in memory
Unrooted stake delegation fork deltas currently spilled to disk
Rooted stake delegations written to disk because the in-memory root pool was full
Stake delegation fork deltas written to disk because the in-memory delta pool was full
Reads from the stake delegation spill file by all tiles
Writes to the stake delegation spill file by all tiles
Time all tiles spent in stake delegation spill file reads and writes
Accounts not found in the account database cache and read from disk, by cache size class
Used in:
0-128 B
129-512 B
513 B-2 KiB
2-8 KiB
8-32 KiB
32-128 KiB
128 KiB-1 MiB
1-10 MiB
Accounts evicted from the account database cache to make space, by cache size class
Used in:
0-128 B
129-512 B
513 B-2 KiB
2-8 KiB
8-32 KiB
32-128 KiB
128 KiB-1 MiB
1-10 MiB
New account versions committed to the index, by destination cache size class
Used in:
0-128 B
129-512 B
513 B-2 KiB
2-8 KiB
8-32 KiB
32-128 KiB
128 KiB-1 MiB
1-10 MiB
Account versions committed over an existing version of the same fork and generation, by destination cache size class
Used in:
0-128 B
129-512 B
513 B-2 KiB
2-8 KiB
8-32 KiB
32-128 KiB
128 KiB-1 MiB
1-10 MiB
Readonly accounts acquired from the account database, by cache size class of the account's current data size
Used in:
0-128 B
129-512 B
513 B-2 KiB
2-8 KiB
8-32 KiB
32-128 KiB
128 KiB-1 MiB
1-10 MiB
Writable accounts acquired from the account database, by cache size class of the account's current data size
Used in:
0-128 B
129-512 B
513 B-2 KiB
2-8 KiB
8-32 KiB
32-128 KiB
128 KiB-1 MiB
1-10 MiB
Replay loop iterations that processed no FEC set, by reason
Used in:
The scheduler was full
The reassembly queue was empty
Waiting for PoH to end our leader block
The bank pool was full
A block's first FEC set waited for a staging lane
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Program cache value capacity, by size class
Used in:
0-128 KiB
128-512 KiB
512 KiB-1 MiB
1-2 MiB
2-4 MiB
4-11 MiB
Program cache value slots in use, by size class
Used in:
0-128 KiB
128-512 KiB
512 KiB-1 MiB
1-2 MiB
2-4 MiB
4-11 MiB
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Blocks added to the replay scheduler, by staging state
Used in:
Added as staged
Added as unstaged
Added as the dead child of a dead block, never dispatched
Times the replay scheduler stopped dispatching a block, by reason
Used in:
A block fully replayed and the block activated next, if any, was not its child
A partially replayed block ran out of work to dispatch
A block was abandoned, other than by minority fork pruning
A block was pruned for being on a minority fork (not also counted as Abandoned)
A block that could only speculate on out-of-band lthash additions gave way to another lane head's real work
A partially replayed block owing out-of-band lthash additions had nothing to dispatch because the speculation pool had no free entry
Out-of-band lthash tasks handed to exec tiles, each a batch of the accounts SchedLthashSubDispatched and SchedLthashAddDispatched count, by type
Used in:
Subtraction tasks, each a batch of accounts hashed on the parent fork
Addition tasks, each a batch of accounts hashed on the block's fork
Out-of-band lthash work discarded when a block was abandoned, demoted, or switched to in-band hashing, by type
Used in:
Additions owed by abandoned blocks, never created
Pseudo transactions of abandoned blocks, never surfaced
Additions in the delta of an abandoned or demoted block, net of undone
Subtractions of abandoned blocks, never dispatched
Subtractions in the delta of an abandoned or demoted block
Additions of abandoned blocks that were batched behind a transaction and never reached a tile
Additions owed by blocks that picked in-band hashing, never created
Subtractions of blocks that picked in-band hashing, never dispatched
Writes the replay scheduler saw for out-of-band lthash, by result
Used in:
First write to the account since its last addition pseudo transaction, which makes the block owe it an addition
Write to an account the block already owed an addition for
Writes the replay scheduler saw for out-of-band lthash, the same as SchedLthashWriteObserved, by type
Used in:
Declared writable accounts of parsed transactions, as handed to the dispatcher
Accounts registered by start-of-block processing
Writable lookup table accounts registered at transaction completion
Accounts re-registered when their block was demoted
Accounts re-registered when a speculative addition was given up
Bank pool index at the head of each replay staging lane, ULONG_MAX if the lane is empty
Used in:
Staging lane 0
Staging lane 1
Staging lane 2
Staging lane 3
Transactions with address lookup tables the replay scheduler tried to resolve, by result
Used in:
Address lookup tables expanded successfully
Address lookup table expansion failed
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Duration of ingesting one queried FEC set into the scheduler
Used in:
[0, 10) nanos
[10, 23) nanos
[23, 52) nanos
[52, 118) nanos
[118, 269) nanos
[269, 612) nanos
[612, 1392) nanos
[1392, 3167) nanos
[3167, 7206) nanos
[7206, 16396) nanos
[16396, 37307) nanos
[37307, 84887) nanos
[84887, 193150) nanos
[193150, 439488) nanos
[439488, 1000000) nanos
[1000000, inf) nanos
Sum of samples (nanos)
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Metrics of one resolv tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Transactions dropped, by reason
Transactions with an unknown blockhash added to the stash
Stashed transactions removed, by reason
Address lookup table resolution attempts, by result
Readonly accounts acquired from the account database, by cache size class of the account's current data size
Accounts not found in the account database cache and read from disk, by cache size class
Accounts that had to wait for another reader to finish loading them into the cache
Bytes read from the account database file
Reads from the account database file
Account data bytes copied in memory between cache lines, staging buffers and callers; disk reads excluded
Accounts not found in the account database cache and read from disk, by cache size class
Used in:
0-128 B
129-512 B
513 B-2 KiB
2-8 KiB
8-32 KiB
32-128 KiB
128 KiB-1 MiB
1-10 MiB
Readonly accounts acquired from the account database, by cache size class of the account's current data size
Used in:
0-128 B
129-512 B
513 B-2 KiB
2-8 KiB
8-32 KiB
32-128 KiB
128 KiB-1 MiB
1-10 MiB
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Stashed transactions removed, by reason
Used in:
Published once its blockhash became known
Evicted as the oldest entry to make room for a new one
Removed without publishing because LUT resolution failed or no bank was available
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Transactions dropped, by reason
Used in:
No bank was available
Blockhash expired
Another transaction in the bundle failed
Address lookup table resolution attempts, by result
Used in:
The transaction referenced an index in a LUT that didn't exist
The account referenced as a LUT hasn't been initialized
The account referenced as a LUT couldn't be parsed
The account referenced as a LUT wasn't owned by the ALUT program ID
The account referenced as a LUT couldn't be found
Resolved successfully
Metrics of one rotor tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Network packets sent, including requests, pings and pongs
Signed repair messages queued, by type; window types send one per missing shred (up to 32), others one per attempt
Highest slot fully delivered to replay, 0 if none
Highest slot a shred was received for, 0 if none
First turbine slot seen, the catchup target, 0 if none
Blocks in the eager, notar and final block treaps, waiting for discovery
Pending request attempts awaiting a response; a hedge is another attempt
FEC sets published to replay
Shreds received at or below the root, ignored
Data shreds received as repair responses, by the pending request they matched
Alpenglow repair metadata responses received
Metadata responses that failed to decode
Metadata responses matching no outstanding request
ParentFecSetCount responses accepted
ParentFecSetCount responses that failed verification
FecSetRoot responses accepted
FecSetRoot responses that failed verification
Root advanced messages received from replay
Missing FEC messages received from replay, each a from-root reconsume
Pings rejected, by reason
Time from sending a repair request to receiving its response
Delay scheduled before a request's next attempt: its peer's hedge time, 16.8 ms to 100 ms, 16.8 ms after an escape
Timeouts in the eager, notar and final timeout queues
Timeouts dropped because their block is gone or their work is done
Timeouts deferred because no peer could be picked
Request attempts after the first for the same timer
Pending requests with no response after about 1 s (PENDING_TTL); a peer is banned for 10 s after 32 in a row, pending-pool evictions not counted
Requests sent to elicit a ping from a peer with a new address
Cluster-wide wait for turbine before a missing FEC set is repaired (srtt+4*rttvar, clamped to 20-250 ms; 250 ms until sampled)
Messages in the sign pool waiting for a signature from the sign tiles
Highest slot past the window with a verified cert, the staked catchup target, 0 if none
WindowIndexForBlockId requests that expired unanswered, at or above the root
Retries sent to a random staked peer because 3 attempts in a row brought no new shred of the FEC set
Peers banned for 10 s after 4 WindowIndexForBlockId requests in a row expired, whatever their other replies
Request attempts sent after 3 or more earlier attempts for the same timer
Uncapped hedge time (srtt+4*rttvar, 100 ms if unmeasured) of each staked peer an escaped retry was sent to
Of TimeoutNoPeerDeferred, retries of a block not known only from turbine, which pick only staked peers: every staked peer was banned, without an address or at its in-flight limit
Timeouts put back without sending a request, by reason, one per deferral
Timeouts dropped because their role's timeout queue was full; the block goes back to its treap and the work is discovered again
Identity switches that halted signing; no request, ping or pong is signed until the switch completes and unhalts
Time the earliest due timeout has waited past its due time when metrics were written, 0 if none is due; grows while signing is halted or fewer than 40 sign slots are free
First certified slot in the window seen by a staked node, which starts its catchup, 0 if none yet
Blocks whose discovery stopped because their role's timeout queue filled; the block goes back to its treap and its remaining FEC sets are queued later
Uncapped hedge time (srtt+4*rttvar, 100 ms if unmeasured) of each staked peer an escaped retry was sent to
Used in:
[0, 10000000) nanos
[10000000, 16378937) nanos
[16378937, 26826958) nanos
[26826958, 43939706) nanos
[43939706, 71968568) nanos
[71968568, 117876865) nanos
[117876865, 193069775) nanos
[193069775, 316227769) nanos
[316227769, 517947472) nanos
[517947472, 848342904) nanos
[848342904, 1389495502) nanos
[1389495502, 2275845935) nanos
[2275845935, 3727593730) nanos
[3727593730, 6105402305) nanos
[6105402305, 10000000000) nanos
[10000000000, inf) nanos
Sum of samples (nanos)
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Pings rejected, by reason
Used in:
Failed to decode
Sender is not a known peer
Signature did not verify
Pong not sent: signing halted for an identity switch, or no free sign slot
Pong not sent: sender address differs from the peer's gossip address
Pong not sent: no request was sent to the peer within the ping TTL
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Signed repair messages queued, by type; window types send one per missing shred (up to 32), others one per attempt
Used in:
Pong
Shred at an index
Highest shred at or above an index
Ancestors of an orphan slot
Parent and FEC set count of a block (Alpenglow)
Merkle root of a FEC set (Alpenglow)
Shred at an index of a specific block (Alpenglow)
Time from sending a repair request to receiving its response
Used in:
[0, 10000000) nanos
[10000000, 13894955) nanos
[13894955, 19306977) nanos
[19306977, 26826958) nanos
[26826958, 37275937) nanos
[37275937, 51794747) nanos
[51794747, 71968568) nanos
[71968568, 100000001) nanos
[100000001, 138949551) nanos
[138949551, 193069775) nanos
[193069775, 268269582) nanos
[268269582, 372759375) nanos
[372759375, 517947471) nanos
[517947471, 719685675) nanos
[719685675, 1000000000) nanos
[1000000000, inf) nanos
Sum of samples (nanos)
Delay scheduled before a request's next attempt: its peer's hedge time, 16.8 ms to 100 ms, 16.8 ms after an escape
Used in:
[0, 10000000) nanos
[10000000, 18615210) nanos
[18615210, 34652605) nanos
[34652605, 64506553) nanos
[64506553, 120080305) nanos
[120080305, 223532013) nanos
[223532013, 416109543) nanos
[416109543, 774596665) nanos
[774596665, 1441927981) nanos
[1441927981, 2684179260) nanos
[2684179260, 4996656140) nanos
[4996656140, 9301380482) nanos
[9301380482, 17314715371) nanos
[17314715371, 32231706785) nanos
[32231706785, 60000000000) nanos
[60000000000, inf) nanos
Sum of samples (nanos)
Data shreds received as repair responses, by the pending request they matched
Used in:
Response to a pending WindowIndexForBlockId request
Response to a pending WindowIndex or HighestWindowIndex request
Matched no pending request: a duplicate, or an answer to an expired or evicted request
Repair response that showed the slot's leader equivocating, dropped before matching; only unverified-nonce HighestWindowIndex or Orphan replies
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Timeouts put back without sending a request, by reason, one per deferral
Used in:
HighestWindowIndex put back: turbine delivered a FEC set of the slot within the eager wait
FEC set request put back: every data shred arrived, the set has not completed
Metrics of one rpc tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Time to handle one HTTP request or WebSocket message, excluding socket I/O
RPC requests handled, by method, before parameter validation
Open HTTP connections to the RPC service
Open WebSocket connections to the RPC service
WebSocket connections subscribed to each notification type
Notifications published to at least one subscriber, by notification type
Notifications sent, counting each subscriber, by notification type
Readonly accounts acquired from the account database, by cache size class of the account's current data size
Accounts not found in the account database cache and read from disk, by cache size class
Accounts that had to wait for another reader to finish loading them into the cache
Bytes read from the account database file
Reads from the account database file
Account data bytes copied in memory between cache lines, staging buffers and callers; disk reads excluded
Accounts not found in the account database cache and read from disk, by cache size class
Used in:
0-128 B
129-512 B
513 B-2 KiB
2-8 KiB
8-32 KiB
32-128 KiB
128 KiB-1 MiB
1-10 MiB
Readonly accounts acquired from the account database, by cache size class of the account's current data size
Used in:
0-128 B
129-512 B
513 B-2 KiB
2-8 KiB
8-32 KiB
32-128 KiB
128 KiB-1 MiB
1-10 MiB
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Time to handle one HTTP request or WebSocket message, excluding socket I/O
Used in:
[0, 1000) nanos
[1000, 2276) nanos
[2276, 5180) nanos
[5180, 11789) nanos
[11789, 26830) nanos
[26830, 61060) nanos
[61060, 138962) nanos
[138962, 316253) nanos
[316253, 719735) nanos
[719735, 1637987) nanos
[1637987, 3727764) nanos
[3727764, 8483720) nanos
[8483720, 19307419) nanos
[19307419, 43940208) nanos
[43940208, 100000000) nanos
[100000000, inf) nanos
Sum of samples (nanos)
RPC requests handled, by method, before parameter validation
Used in:
GET /genesis.tar.bz2
Unknown method
getAccountInfo
getBalance
getBlockHeight
getClusterNodes
getEpochInfo
getGenesisHash
getHealth
getIdentity
getInflationGovernor
getLatestBlockhash
getMinimumBalanceForRentExemption
getMultipleAccounts
getSlot
getTransactionCount
getVersion
getMaxRetransmitSlot
getSlotLeader
getSlotLeaders
getLeaderSchedule
getAgGenesisCert
slotSubscribe
slotUnsubscribe
voteSubscribe
voteUnsubscribe
Known method not implemented by Firedancer
Malformed JSON-RPC request
GET /health
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Notifications published to at least one subscriber, by notification type
Used in:
vote
slot
Notifications sent, counting each subscriber, by notification type
Used in:
vote
slot
WebSocket connections subscribed to each notification type
Used in:
vote
slot
Metrics of one rserve tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Requests with a supported tag and correct size (pongs on tag alone), by type, before identity, signature and ping checks
Bytes of incoming repair request payloads
Packets dropped before parsing, by reason
Requests dropped during validation, by reason
Valid requests that could not be answered, by type
Network packets sent, including pings and responses
Response packets sent, by type; Orphan sends up to 11 per request
Payload bytes sent in response packets
Store reads for repair responses, by result
Shred bytes read from the store for repair responses
Shreds currently in the shreds database
Shreds the shreds database can hold
Logical bytes occupied by live wire shreds and spilled FEC payloads
Logical wire-ring high-water plus spill pages high-water
Duration of persisting one accepted data shred
Data shreds persisted to the repair store
Data shreds rejected by the repair store API
Bytes written by persisted data shreds
Duration of preevicting one FEC payload from the store RAM cache
FEC payloads preevicted from the store RAM cache to disk
FEC payload bytes preevicted from the store RAM cache to disk
Immediately available FEC payload cache entries
FEC payload cache capacity
Free entries background preeviction tries to restore
Free entry count below which background preeviction starts
Active entries in the ping cache
Ping cache entry capacity
Ping cache entries evicted for a new peer (cache full); TTL expiries not counted
Store reads for repair responses, by result
Used in:
Read served from the store
Not in the store
Index binding or ring cell changed during the read
Cached highest-shred bound was no longer readable
Duration of persisting one accepted data shred
Used in:
[0, 100) nanos
[100, 268) nanos
[268, 719) nanos
[719, 1929) nanos
[1929, 5175) nanos
[5175, 13884) nanos
[13884, 37250) nanos
[37250, 99939) nanos
[99939, 268129) nanos
[268129, 719371) nanos
[719371, 1930022) nanos
[1930022, 5178115) nanos
[5178115, 13892523) nanos
[13892523, 37272675) nanos
[37272675, 100000000) nanos
[100000000, inf) nanos
Sum of samples (nanos)
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Requests dropped during validation, by reason
Used in:
Signature failed to verify
Sent by our own identity
Addressed to another identity
Timestamp outside the accepted window
Pong token outdated or invalid
Shred index past the end of a block
FEC set index unaligned or past the end of the block
Packets dropped before parsing, by reason
Used in:
Packet shorter than a message tag
Ping sent to the repair server port
Unknown or unsupported message tag
Message size does not match its tag
Ancestor hashes request, which is not served
Requests with a supported tag and correct size (pongs on tag alone), by type, before identity, signature and ping checks
Used in:
Pong
Shred at an index
Highest shred at or above an index
Ancestors of an orphan slot
Parent and FEC set count of a block (Alpenglow)
Merkle root of a FEC set (Alpenglow)
Shred at an index of a specific block (Alpenglow)
Valid requests that could not be answered, by type
Used in:
Pong
Shred at an index
Highest shred at or above an index
Ancestors of an orphan slot
Parent and FEC set count of a block (Alpenglow)
Merkle root of a FEC set (Alpenglow)
Shred at an index of a specific block (Alpenglow)
Response packets sent, by type; Orphan sends up to 11 per request
Used in:
Ping sent to a requester not yet in the ping cache
Shred at an index
Highest shred at or above an index
Shred from an ancestor of an orphan slot
Parent and FEC set count of a block (Alpenglow)
Merkle root of a FEC set (Alpenglow)
Shred at an index of a specific block (Alpenglow)
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Duration of preevicting one FEC payload from the store RAM cache
Used in:
[0, 100) nanos
[100, 268) nanos
[268, 719) nanos
[719, 1929) nanos
[1929, 5175) nanos
[5175, 13884) nanos
[13884, 37250) nanos
[37250, 99939) nanos
[99939, 268129) nanos
[268129, 719371) nanos
[719371, 1930022) nanos
[1930022, 5178115) nanos
[5178115, 13892523) nanos
[13892523, 37272675) nanos
[37272675, 100000000) nanos
[100000000, inf) nanos
Sum of samples (nanos)
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Why a metrics sweep was taken
Used in: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Periodic sweep
Our leader slot started
Our leader slot ended
Our leader slot ended and the next one started
Snapshot load changed state
Replay finished loading and started processing
Replay caught up to the cluster tip
Metrics of one shred tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Shred destinations in each cluster contact info update (Frankendancer only; on Firedancer see the gossip tile peer count)
Microblock batches abandoned because we switched slots without finishing the current slot
Leader blocks whose parent block_id was unknown, so the chained merkle root was zeroed
Size of each microblock batch that is shredded
Microblocks in each microblock batch that is shredded
Duration of shredding one microblock batch
Duration of verifying and processing one shred received from the network
Duration of synchronously spilling one FEC payload to the store
Shreds assigned to this tile that passed the initial parse and chained check, by processing result, plus received frags that failed the initial parse; InitialRejected and FecFatalRejected are subsets of Rejected
Incomplete FEC sets evicted from the resolver to make room for new ones
Shreds rejected before any resources were allocated for the FEC set
Shreds rejected because they're not chained merkle shreds
FEC sets rejected for reasons that cause the whole FEC set to become invalid
Shreds that looked like repair responses but had an incorrect nonce
Repair shreds received
Bytes received from network packets with repair shreds, including network headers
Turbine shreds received
Bytes received from network packets with turbine shreds, including network headers
FEC sets that completed with at least one repair shred contributing
Estimated time repair advanced FEC set completion: the delay from a repair-assisted completion until turbine had delivered as many shreds for that set as repair contributed. Counts arrivals rather than distinct indices, so turbine duplicates bias this low
FEC payloads the shred tile spilled to the store synchronously because the store RAM cache was full
FEC payload bytes the shred tile spilled to the store synchronously
Duration of verifying and processing one shred received from the network
Used in:
[0, 10) nanos
[10, 23) nanos
[23, 52) nanos
[52, 118) nanos
[118, 269) nanos
[269, 612) nanos
[612, 1392) nanos
[1392, 3167) nanos
[3167, 7206) nanos
[7206, 16396) nanos
[16396, 37307) nanos
[37307, 84887) nanos
[84887, 193150) nanos
[193150, 439488) nanos
[439488, 1000000) nanos
[1000000, inf) nanos
Sum of samples (nanos)
Duration of shredding one microblock batch
Used in:
[0, 10000) nanos
[10000, 16379) nanos
[16379, 26827) nanos
[26827, 43940) nanos
[43940, 71969) nanos
[71969, 117878) nanos
[117878, 193071) nanos
[193071, 316230) nanos
[316230, 517951) nanos
[517951, 848348) nanos
[848348, 1389502) nanos
[1389502, 2275854) nanos
[2275854, 3727603) nanos
[3727603, 6105410) nanos
[6105410, 10000000) nanos
[10000000, inf) nanos
Sum of samples (nanos)
Size of each microblock batch that is shredded
Used in:
[0, 1024) bytes
[1024, 1378) bytes
[1378, 1855) bytes
[1855, 2497) bytes
[2497, 3361) bytes
[3361, 4523) bytes
[4523, 6087) bytes
[6087, 8192) bytes
[8192, 11026) bytes
[11026, 14840) bytes
[14840, 19973) bytes
[19973, 26881) bytes
[26881, 36179) bytes
[36179, 48693) bytes
[48693, 65536) bytes
[65536, inf) bytes
Sum of samples (bytes)
Shred destinations in each cluster contact info update (Frankendancer only; on Firedancer see the gossip tile peer count)
Used in:
[0, 1)
[1, 2)
[2, 4)
[4, 9)
[9, 19)
[19, 41)
[41, 88)
[88, 189)
[189, 406)
[406, 873)
[873, 1878)
[1878, 4040)
[4040, 8690)
[8690, 18691)
[18691, 40200)
[40200, inf)
Sum of samples
Microblocks in each microblock batch that is shredded
Used in:
[0, 2)
[2, 3)
[3, 4)
[4, 6)
[6, 8)
[8, 11)
[11, 16)
[16, 23)
[23, 32)
[32, 45)
[45, 64)
[64, 91)
[91, 128)
[128, 181)
[181, 256)
[256, inf)
Sum of samples
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Shreds assigned to this tile that passed the initial parse and chained check, by processing result, plus received frags that failed the initial parse; InitialRejected and FecFatalRejected are subsets of Rejected
Used in:
Shred was for a slot whose leader is unknown
Shred parsing failed
Shred was equivocated with another shred
Shred was invalid for one of many reasons
Shred was ignored because we had already reconstructed the FEC set, or was for an old slot
Shred was a duplicate of a previously received valid shred of an incomplete FEC set
Shred accepted into an incomplete FEC set
Shred accepted and resulted in a valid, complete FEC set
Turbine shred for a slot we led, echoed back to us by a relay
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Estimated time repair advanced FEC set completion: the delay from a repair-assisted completion until turbine had delivered as many shreds for that set as repair contributed. Counts arrivals rather than distinct indices, so turbine duplicates bias this low
Used in:
[0, 100000) nanos
[100000, 227585) nanos
[227585, 517948) nanos
[517948, 1178770) nanos
[1178770, 2682699) nanos
[2682699, 6105409) nanos
[6105409, 13894969) nanos
[13894969, 31622805) nanos
[31622805, 71968623) nanos
[71968623, 163789476) nanos
[163789476, 372759564) nanos
[372759564, 848343226) nanos
[848343226, 1930698226) nanos
[1930698226, 4393971126) nanos
[4393971126, 10000000000) nanos
[10000000000, inf) nanos
Sum of samples (nanos)
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Duration of synchronously spilling one FEC payload to the store
Used in:
[0, 100) nanos
[100, 268) nanos
[268, 719) nanos
[719, 1929) nanos
[1929, 5175) nanos
[5175, 13884) nanos
[13884, 37250) nanos
[37250, 99939) nanos
[99939, 268129) nanos
[268129, 719371) nanos
[719371, 1930022) nanos
[1930022, 5178115) nanos
[5178115, 13892523) nanos
[13892523, 37272675) nanos
[37272675, 100000000) nanos
[100000000, inf) nanos
Sum of samples (nanos)
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Metrics of one sign tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Duration of signing a message
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Duration of signing a message
Used in:
[0, 10) nanos
[10, 23) nanos
[23, 52) nanos
[52, 118) nanos
[118, 269) nanos
[269, 612) nanos
[612, 1392) nanos
[1392, 3167) nanos
[3167, 7206) nanos
[7206, 16396) nanos
[16396, 37307) nanos
[37307, 84887) nanos
[84887, 193150) nanos
[193150, 439488) nanos
[439488, 1000000) nanos
[1000000, inf) nanos
Sum of samples (nanos)
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Metrics of one snapct tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Snapshot load controller state: 0=init (Initializing), 1=waiting_for_peers, 2=waiting_for_peers_incremental (Waiting for peers (incremental)), 3=collecting_peers, 4=collecting_peers_incremental (Collecting peers (incremental)), 5=reading_full_file (Reading full snapshot file), 6=flushing_full_file_fini (Full file read, draining downstream), 7=flushing_full_file_done (Full file loaded, draining downstream), 8=flushing_full_file_reset (Full file load aborted, resetting downstream), 9=reading_incremental_file (Reading incremental snapshot file), 10=flushing_incremental_file_fini (Incremental file read, draining downstream), 11=flushing_incremental_file_done (Incremental file loaded, draining downstream), 12=flushing_incremental_file_reset (Incremental file load aborted, resetting downstream), 13=reading_full_http (Downloading full snapshot), 14=flushing_full_http_fini (Full download read, draining downstream), 15=flushing_full_http_done (Full download loaded, draining downstream), 16=flushing_full_http_reset (Full download aborted, resetting downstream), 17=reading_incremental_http (Downloading incremental snapshot), 18=flushing_incremental_http_fini (Incremental download read, draining downstream), 19=flushing_incremental_http_done (Incremental download loaded, draining downstream), 20=flushing_incremental_http_reset (Incremental download aborted, resetting downstream), 21=shutdown (Snapshot load complete)
Snapshot file bytes read (download or local file) in the current attempt, by type; resets on retry
Size of each snapshot file being loaded; may change if the load restarts
Bytes of each downloaded snapshot saved to the snapshot directory so far; resets if the load restarts
Snapshot downloads retried, by snapshot type
Slot of the incremental snapshot replay is predicted to start from, 0 if none chosen; may change if the load restarts
Snapshot downloads retried, by snapshot type
Used in:
Full snapshot
Incremental snapshot
Snapshot load state
Used in: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Initializing
Waiting for peers
Waiting for peers (incremental)
Collecting peers
Collecting peers (incremental)
Reading full snapshot file
Full file read, draining downstream
Full file loaded, draining downstream
Full file load aborted, resetting downstream
Reading incremental snapshot file
Incremental file read, draining downstream
Incremental file loaded, draining downstream
Incremental file load aborted, resetting downstream
Downloading full snapshot
Full download read, draining downstream
Full download loaded, draining downstream
Full download aborted, resetting downstream
Downloading incremental snapshot
Incremental download read, draining downstream
Incremental download loaded, draining downstream
Incremental download aborted, resetting downstream
Snapshot load complete
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Size of each snapshot file being loaded; may change if the load restarts
Used in:
Full snapshot
Incremental snapshot
Snapshot file bytes read (download or local file) in the current attempt, by type; resets on retry
Used in:
Full snapshot
Incremental snapshot
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Bytes of each downloaded snapshot saved to the snapshot directory so far; resets if the load restarts
Used in:
Full snapshot
Incremental snapshot
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Metrics of one snapdc tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Snapshot decompressor state: 0=idle, 1=processing, 2=finishing, 3=error, 4=shutdown
Compressed bytes of each snapshot read in the current attempt; sum across snapdc tiles
Decompressed bytes of each snapshot produced in the current attempt; sum across snapdc tiles
Compressed bytes of each snapshot read in the current attempt; sum across snapdc tiles
Used in:
Full snapshot
Incremental snapshot
Decompressed bytes of each snapshot produced in the current attempt; sum across snapdc tiles
Used in:
Full snapshot
Incremental snapshot
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Snapshot decompressor state: 0=idle, 1=processing, 2=finishing, 3=error, 4=shutdown
Used in:
Idle
Processing
Finishing
Error
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Metrics of one snapin tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Snapshot inserter state: 0=idle, 1=processing, 2=finishing, 3=error, 4=shutdown
Decompressed bytes of each snapshot this tile parsed in the current attempt; every snapin tile parses the whole stream, so take the min across tiles
Bytes this tile has written to the accounts database file
Accounts this tile processed in the current load, by result (Ignored accounts are not inserted); sum across tiles, rewinds on retry
Accounts processed across all snapshots
Account batches processed across all snapshots
Accounts this tile processed in the current load, by result (Ignored accounts are not inserted); sum across tiles, rewinds on retry
Used in:
New account inserted
Replaced an older duplicate inserted earlier
Dropped because an inserted duplicate was newer
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Decompressed bytes of each snapshot this tile parsed in the current attempt; every snapin tile parses the whole stream, so take the min across tiles
Used in:
Full snapshot
Incremental snapshot
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Snapshot inserter state: 0=idle, 1=processing, 2=finishing, 3=error, 4=shutdown
Used in:
Idle
Processing
Finishing
Error
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Metrics of one snapld tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Snapshot loader state: 0=idle, 1=processing, 2=finishing, 3=error, 4=shutdown
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Snapshot loader state: 0=idle, 1=processing, 2=finishing, 3=error, 4=shutdown
Used in:
Idle
Processing
Finishing
Error
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Metrics of one snapmk tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Snapshot creation job state: 0=idle (Idle, waiting for a job), 1=start (Starting), 2=tar_headers (Writing tar headers), 3=manifest (Writing manifest), 4=accdb_cache (Writing cached accounts), 5=accdb_cache_flush (Flushing cached accounts), 6=accdb_cache_finish (Waiting for cached account flush), 7=accdb_disk (Writing on-disk accounts), 8=accdb_disk_flush (Flushing on-disk accounts), 9=accdb_disk_finish (Waiting for on-disk account flush), 10=accdb_delta (Writing incremental accounts), 11=accdb_delta_flush (Flushing incremental accounts), 12=accdb_delta_finish (Waiting for incremental account flush), 13=status_cache (Writing status cache), 14=eof_marker (Writing tar EOF marker), 15=done (Done, notifying replay), 16=fail (Failed, cleaning up), 17=sleep (Sleeping), 18=startup (Waiting for startup), 19=startup_burst (Publishing pre-existing snapshots)
Snapshot creation jobs started, by snapshot type
Snapshot creation jobs finished successfully, by snapshot type
Slot of the most recently started snapshot creation, by snapshot type, 0 if none since boot
Slot of the most recently finished snapshot creation, by snapshot type, 0 if none since boot
Uncompressed bytes fed to the compressor
Compressed bytes written to snapshot files
Bytes of the accounts database cache region scanned so far for the snapshot being created. Resets when a snapshot starts and holds at the total once the cached accounts phase finishes. Stays zero for an incremental snapshot, which does not scan the cache
Size of the accounts database cache region a cached accounts phase scans
Accounts changed since the base snapshot that have been written so far for the incremental snapshot being created. Resets when a snapshot starts, and stays zero for a full snapshot, which has no changed account walk
Accounts changed since the base snapshot, sampled when the incremental snapshot started; 0 for a full snapshot. Replay may record more changes while the job runs, so progress can exceed this
Cumulative time blocked on I/O
Cumulative time spent compressing data
Distinct accounts changed in rooted slots since the last full snapshot job started; saturates at IncrementalAccountCapacity
Maximum changed accounts that can be tracked; once reached, no incremental snapshot can be created until the next full snapshot
Logical bytes allocated across managed snapshot files, including partial and retained files
Snapshot creation job state: 0=idle (Idle, waiting for a job), 1=start (Starting), 2=tar_headers (Writing tar headers), 3=manifest (Writing manifest), 4=accdb_cache (Writing cached accounts), 5=accdb_cache_flush (Flushing cached accounts), 6=accdb_cache_finish (Waiting for cached account flush), 7=accdb_disk (Writing on-disk accounts), 8=accdb_disk_flush (Flushing on-disk accounts), 9=accdb_disk_finish (Waiting for on-disk account flush), 10=accdb_delta (Writing incremental accounts), 11=accdb_delta_flush (Flushing incremental accounts), 12=accdb_delta_finish (Waiting for incremental account flush), 13=status_cache (Writing status cache), 14=eof_marker (Writing tar EOF marker), 15=done (Done, notifying replay), 16=fail (Failed, cleaning up), 17=sleep (Sleeping), 18=startup (Waiting for startup), 19=startup_burst (Publishing pre-existing snapshots)
Used in:
Idle, waiting for a job
Starting
Writing tar headers
Writing manifest
Writing cached accounts
Flushing cached accounts
Waiting for cached account flush
Writing on-disk accounts
Flushing on-disk accounts
Waiting for on-disk account flush
Writing incremental accounts
Flushing incremental accounts
Waiting for incremental account flush
Writing status cache
Writing tar EOF marker
Done, notifying replay
Failed, cleaning up
Sleeping
Waiting for startup
Publishing pre-existing snapshots
Slot of the most recently finished snapshot creation, by snapshot type, 0 if none since boot
Used in:
Full snapshot
Incremental snapshot
Slot of the most recently started snapshot creation, by snapshot type, 0 if none since boot
Used in:
Full snapshot
Incremental snapshot
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Snapshot creation jobs finished successfully, by snapshot type
Used in:
Full snapshot
Incremental snapshot
Snapshot creation jobs started, by snapshot type
Used in:
Full snapshot
Incremental snapshot
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Metrics of one snaprd tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Bytes read from the account database file
Account database bytes read so far for the current snapshot creation job
Account database bytes to read for the current snapshot creation job
Cumulative time blocked on I/O
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Metrics of one snapsv tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Snapshot files currently available for download
HTTP requests served
Open HTTP connections to the snapshot service
Bytes received in requests to the snapshot service
Bytes sent in responses from the snapshot service
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Snapshot files currently available for download
Used in:
Full snapshot
Incremental snapshot
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Metrics of one snapzp tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Accounts compressed
Uncompressed bytes fed to the compressor
Compressed bytes written to snapshot files
Cumulative time blocked on I/O
Cumulative time spent compressing accounts
Cached account reads torn by eviction, handed back to snapmk to un-claim
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Metrics of one sock tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
sendmmsg syscalls dispatched, by result
recvmmsg syscalls dispatched
Packets received
Packets sent
Packets that failed to send
Bytes of packets successfully sent
UDP payload bytes received plus 42 bytes of synthesized Ethernet/IPv4/UDP headers per packet
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
sendmmsg syscalls dispatched, by result
Used in:
Success
ENOBUFS, EAGAIN error
EPERM error (blocked by netfilter)
ENETUNREACH, EHOSTUNREACH error
ENONET, ENETDOWN, EHOSTDOWN error
Unrecognized error code
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Metrics of one verify tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Transactions processed, by result
Gossip vote transactions handled by this verify tile
Our own vote transactions received from txsend
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Transactions processed, by result
Used in:
Transaction verified successfully
Another transaction in the bundle failed
Transaction failed to parse
Transaction was a duplicate
Transaction failed signature verification
Metrics of one votor tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
Inbound datagrams dispatched, by result
Inbound votes processed, by result
Inbound certificates processed, by result
Slots tracked by the voting state machine
Free entries in the voting state machine's slot table
Highest slot with a final or fast-final cert seen by votor, 0 if none
Slots tracked by the vote and cert pool
Free entries in the vote and cert pool's slot table
Highest slot the pool has finalized (slow-final also needs the slot notarized), 0 if none
Rank of this validator in the current epoch (0 is the highest rank), ULONG_MAX if unranked (unstaked)
Peers with an open outbound connection; replaced by ClientConnUsed
Times the vote history file was written
Root of the last written vote history file, which votor waits to vote past, 0 if not written yet
Size of the last written vote history file
Vote history writes skipped because the history was too large; the file is emptied until it fits
Aggregate BLS signature checks of votes, certificates and reward certificates; a search for bad signers counts once
First slot votor may vote in, past the vote history file and the votes it sent
Leader windows given up because certificates already let a later window start, eg. while catching up after a restart
Certificates and standstill votes not sent because we are not staked in their epoch
UDP datagrams received (one may carry several coalesced QUIC packets)
Bytes of UDP datagrams received, including Ethernet, IPv4 and UDP headers
UDP datagrams sent
Bytes of UDP datagrams sent, including Ethernet, IPv4 and UDP headers
Connection slots in use, by state; sum + ConnFree = ConnCapacity
Connection slots free
Connection slots
Connections created
Connections gracefully closed
Connections aborted
Connections timed out
Connections that failed to create due to lack of slots
Retries rejected: an invalid or expired token as server, a late or invalid Retry packet as client
Packets that failed decryption, by encryption level
Packets dropped because the decryption key was missing, by encryption level
Received packets dropped, by reason
Packets with an unknown connection ID, by packet type
QUIC packets retransmitted, by encryption level
Handshake flows created
Handshakes dropped due to alloc fail
Handshakes dropped due to eviction
Stream receive events
Stream payload bytes received
QUIC frames received, by frame type
QUIC frames that failed to parse
Attempts to acquire packet metadata for transmission, by result
Received packets processed for acknowledgement, by result
Duration of one fd_quic_service call that processed a due connection timer
Duration of processing one received packet
UDP datagrams received (one may carry several coalesced QUIC packets)
Bytes of UDP datagrams received, including Ethernet, IPv4 and UDP headers
UDP datagrams sent
Bytes of UDP datagrams sent, including Ethernet, IPv4 and UDP headers
Connection slots in use, by state; sum + ConnFree = ConnCapacity
Connection slots free
Connection slots
Connections created
Connections gracefully closed
Connections aborted
Connections timed out
Connections that failed to create due to lack of slots
Retries rejected: an invalid or expired token as server, a late or invalid Retry packet as client
Packets that failed decryption, by encryption level
Packets dropped because the decryption key was missing, by encryption level
Received packets dropped, by reason
Packets with an unknown connection ID, by packet type
QUIC packets retransmitted, by encryption level
Handshake flows created
Handshakes dropped due to alloc fail
Handshakes dropped due to eviction
Stream receive events
Stream payload bytes received
QUIC frames received, by frame type
QUIC frames that failed to parse
Attempts to acquire packet metadata for transmission, by result
Received packets processed for acknowledgement, by result
Duration of one fd_quic_service call that processed a due connection timer
Duration of processing one received packet
Initials with an expired Retry token, also counted in ServerConnRetryRejected. Each gets an INVALID_TOKEN close so the client reconnects
Inbound certificates processed, by result
Used in:
Cert was handed to the pool and accepted
Cert was truncated or had trailing bytes
Cert had an invalid version, tag, bitmap, or signature encoding
Cert was for a different shred version
Sending connection had no authenticated identity
Sender was not in the peer set
Sender was not a ranked validator in the cert slot's epoch
Cert slot was either too old or too far in the future
Cert was already in the pool
Cert failed the stake threshold or the aggregate signature check
Sender is banned for a failed signature verification
Connection slots in use, by state; sum + ConnFree = ConnCapacity
Used in:
Handshaking peer
Handshake complete, confirming with peer
Active connection
Peer requested close
Connection terminating due to error
Connection is closing
Connection about to be freed
QUIC frames received, by frame type
Used in:
Unknown frame type
ACK frame
RESET_STREAM frame
STOP_SENDING frame
CRYPTO frame
NEW_TOKEN frame
STREAM frame
MAX_DATA frame
MAX_STREAM_DATA frame
MAX_STREAMS frame
DATA_BLOCKED frame
STREAM_DATA_BLOCKED frame
STREAMS_BLOCKED(bidi) frame
NEW_CONN_ID frame
RETIRE_CONN_ID frame
PATH_CHALLENGE frame
PATH_RESPONSE frame
CONN_CLOSE(transport) frame
CONN_CLOSE(app) frame
HANDSHAKE_DONE frame
PING frame
PADDING frame
DATAGRAM frame
Received packets processed for acknowledgement, by result
Used in:
Packet had no packet number to acknowledge
New ACK range
Merged into existing ACK range
Out of buffers
ACK suppressed by handler
Packets that failed decryption, by encryption level
Used in:
Initial
Early data
Handshake
App data
Received packets dropped, by reason
Used in:
QUIC packet too small
QUIC packet too large
Wrong source IP
Invalid QUIC header
Invalid IPv4 or UDP header
Version negotiation or unsupported version
Packet number already received or too old
Attempts to acquire packet metadata for transmission, by result
Used in:
Success
Packet metadata pool empty
Connection reached its in-flight frame limit
Packets with an unknown connection ID, by packet type
Used in:
Initial
Retry
Handshake
1-RTT
Packets dropped because the decryption key was missing, by encryption level
Used in:
Initial
Early data
Handshake
App data
QUIC packets retransmitted, by encryption level
Used in:
Initial
Early data
Handshake
App data
Duration of processing one received packet
Used in:
[0, 10) nanos
[10, 32) nanos
[32, 101) nanos
[101, 319) nanos
[319, 1008) nanos
[1008, 3185) nanos
[3185, 10064) nanos
[10064, 31800) nanos
[31800, 100480) nanos
[100480, 317492) nanos
[317492, 1003197) nanos
[1003197, 3169857) nanos
[3169857, 10015972) nanos
[10015972, 31648020) nanos
[31648020, 100000000) nanos
[100000000, inf) nanos
Sum of samples (nanos)
Duration of one fd_quic_service call that processed a due connection timer
Used in:
[0, 10) nanos
[10, 32) nanos
[32, 101) nanos
[101, 319) nanos
[319, 1008) nanos
[1008, 3185) nanos
[3185, 10064) nanos
[10064, 31800) nanos
[31800, 100480) nanos
[100480, 317492) nanos
[317492, 1003197) nanos
[1003197, 3169857) nanos
[3169857, 10015972) nanos
[10015972, 31648020) nanos
[31648020, 100000000) nanos
[100000000, inf) nanos
Sum of samples (nanos)
Inbound datagrams dispatched, by result
Used in:
Datagram carried a vote and was handed to vote rx
Datagram carried a cert and was handed to cert rx
Datagram arrived before the votor tile was initialized
Datagram was too small to hold a version and a tag
Datagram tag was neither a vote nor a cert
Datagram arrived on a connection not yet admitted, or already closed
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Connection slots in use, by state; sum + ConnFree = ConnCapacity
Used in:
Handshaking peer
Handshake complete, confirming with peer
Active connection
Peer requested close
Connection terminating due to error
Connection is closing
Connection about to be freed
QUIC frames received, by frame type
Used in:
Unknown frame type
ACK frame
RESET_STREAM frame
STOP_SENDING frame
CRYPTO frame
NEW_TOKEN frame
STREAM frame
MAX_DATA frame
MAX_STREAM_DATA frame
MAX_STREAMS frame
DATA_BLOCKED frame
STREAM_DATA_BLOCKED frame
STREAMS_BLOCKED(bidi) frame
NEW_CONN_ID frame
RETIRE_CONN_ID frame
PATH_CHALLENGE frame
PATH_RESPONSE frame
CONN_CLOSE(transport) frame
CONN_CLOSE(app) frame
HANDSHAKE_DONE frame
PING frame
PADDING frame
DATAGRAM frame
Received packets processed for acknowledgement, by result
Used in:
Packet had no packet number to acknowledge
New ACK range
Merged into existing ACK range
Out of buffers
ACK suppressed by handler
Packets that failed decryption, by encryption level
Used in:
Initial
Early data
Handshake
App data
Received packets dropped, by reason
Used in:
QUIC packet too small
QUIC packet too large
Wrong source IP
Invalid QUIC header
Invalid IPv4 or UDP header
Version negotiation or unsupported version
Packet number already received or too old
Attempts to acquire packet metadata for transmission, by result
Used in:
Success
Packet metadata pool empty
Connection reached its in-flight frame limit
Packets with an unknown connection ID, by packet type
Used in:
Initial
Retry
Handshake
1-RTT
Packets dropped because the decryption key was missing, by encryption level
Used in:
Initial
Early data
Handshake
App data
QUIC packets retransmitted, by encryption level
Used in:
Initial
Early data
Handshake
App data
Duration of processing one received packet
Used in:
[0, 10) nanos
[10, 32) nanos
[32, 101) nanos
[101, 319) nanos
[319, 1008) nanos
[1008, 3185) nanos
[3185, 10064) nanos
[10064, 31800) nanos
[31800, 100480) nanos
[100480, 317492) nanos
[317492, 1003197) nanos
[1003197, 3169857) nanos
[3169857, 10015972) nanos
[10015972, 31648020) nanos
[31648020, 100000000) nanos
[100000000, inf) nanos
Sum of samples (nanos)
Duration of one fd_quic_service call that processed a due connection timer
Used in:
[0, 10) nanos
[10, 32) nanos
[32, 101) nanos
[101, 319) nanos
[319, 1008) nanos
[1008, 3185) nanos
[3185, 10064) nanos
[10064, 31800) nanos
[31800, 100480) nanos
[100480, 317492) nanos
[317492, 1003197) nanos
[1003197, 3169857) nanos
[3169857, 10015972) nanos
[10015972, 31648020) nanos
[31648020, 100000000) nanos
[100000000, inf) nanos
Sum of samples (nanos)
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
Inbound votes processed, by result
Used in:
Vote was handed to the pool and accepted
Vote was truncated or had trailing bytes
Vote had an invalid version, tag, or signature encoding
Sending connection had no authenticated identity
Sender was not in the peer set
Sender is banned for a failed signature verification
Vote slot was either too old or too far in the future
Vote block hash was null
Vote signature was the identity point or outside the G2 subgroup
Sender was not a ranked validator in the vote slot's epoch
Vote was for a different shred version
Voter had already cast this vote
Vote constitutes a slashable offence
Metrics of one waker tile instance, one row per sweep
Used in:
Identifier shared by every row of one sweep
Why a metrics sweep was taken
Slot that triggered the sweep, 0 for periodic sweeps
Snapshot load state
Whether replay had caught up to the cluster tip when the sweep was taken (replay IsCaughtUp); set once per run, never cleared
Tile instances in the sweep, to detect partial sweeps
Index of this tile instance among tiles of its kind
Process ID of the tile
Thread ID of the tile. Same as Pid in production, but might differ in development
CPU the tile last ran on
counter: cumulative since tile start, query deltas. Involuntary context switches
counter: cumulative since tile start, query deltas. Voluntary context switches
counter: cumulative since tile start, query deltas. Major page faults
counter: cumulative since tile start, query deltas. Minor page faults
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Last wall-clock time the tile heartbeated
Whether the tile is currently backpressured
counter: cumulative since tile start, query deltas. Times the tile waited for one or more consumers to catch up to resume publishing
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Times the tile tried to park while idle (efficient mode), by result
counter: cumulative since tile start, query deltas. Device IRQs on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. TLB shootdowns on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
counter: cumulative since tile start, query deltas. Local timer interrupts (LOC) on the tile's dedicated CPU, 0 unless it is the only live tile pinned there
epoll_pwait calls on the outer epoll set that returned, including timeouts; EINTR not counted
Client wakes delivered (readiness fseq raised)
Times the tile tried to park while idle (efficient mode), by result
Used in:
Woken by a doorbell ring
Deadline or park cap lapsed
Pre-park recheck found work, did not park
Tile wall time, by regime; regimes are exhaustive and groups Busy, Idle, Backpressure and Housekeeping partition it
Used in:
Caught up + Housekeeping
Processing + Housekeeping
Backpressure + Housekeeping
Caught up + Prefrag
Processing + Prefrag
Backpressure + Prefrag
Caught up + Postfrag
Processing + Postfrag
Caught up + Sleeping
Backpressure + Sleeping
Tile lifecycle state: 0=booting, 1=running, 2=shutdown
Used in:
Booting
Running
Shutdown
Thread wall time since start, by scheduler state; Idle is the remainder, so parts sum to about the lifetime. User, System and Interrupt have 10 ms resolution; Interrupt is 0 unless it is the only live tile pinned to its CPU
Used in:
Wait (task was runnable but not scheduled)
Idle (task was not runnable)
User (task was scheduled and executing in user mode)
System (task was scheduled and executing in kernel mode)
Interrupt (CPU time stolen by hardirq/softirq handlers or a hypervisor; fixed tiles only)
One repair request message this validator sent to a peer, reported once when it settles: its answer arrived, its metadata reply was checked, the deadline passed (PENDING_TTL, about 1 s after req_time), or it was evicted. Each message asks for one thing: a window_index or window_index_for_block_id pick sends one message per missing shred of the FEC set, so one row each, sharing peer, req_time, attempt and pick; the other kinds are one message per pick. A hedge or retry to another peer is another pick. Messages never sent have no row; see the repair tile's Timeout* and SigningHalted metrics, and TimeoutOverdueNanos for sign slot starvation. Emitted by the repair tile.
Used in:
Validator identity of the peer asked.
IPv4 address the request was sent to, stored as an IPv4-mapped IPv6 address.
UDP port the request was sent to: the peer's serve-repair port in gossip when sent.
The peer's stake rank in the current epoch when the message settled, 0 for the most staked, as ranked by the repair strategy (65535 if unstaked or no longer known).
Kind of request.
Slot of the block asked about.
Block id of the block asked about; all zeros for turbine's block until it completes.
For window_index and window_index_for_block_id, the index of the shred the message asked for. For fec_set_root, index of the first shred of the FEC set asked about. For highest_window_index, the shred index sent: the first index past the highest FEC set received; the peer answers only if it holds a shred at or past it, or the slot is complete. 0 for orphan and parent_and_fec_set_count.
When the message was queued for signing; it leaves once its signature returns, so res_time - req_time includes sign latency.
When this message's answer was received off the network (by the shred tile for shreds, the net tile for metadata replies); for bad, when the reply that failed its proof arrived; 0 if none.
How the message settled.
Whether and why settling this message's pick banned the peer for 10 s. A pick settles once at the deadline, so every message of a pick left unanswered then carries the same ban.
Earlier picks for the same request before this one, 0 for the first; at most 255.
How the peer was picked.
Role of the block asked about in its slot when sent.
Whether and why settling this message's pick banned the peer for 10 s. A pick settles once at the deadline, so every message of a pick left unanswered then carries the same ban.
Used in:
Not banned.
Its 4th window_index_for_block_id pick in a row expired with no by-id answer.
Its 32nd pick in a row expired with no answer of any kind.
Its metadata reply failed its proof.
How the message settled.
Used in:
Its shred arrived, or its metadata reply passed its proof. A highest_window_index or orphan reply that does not complete the slot carries a nonce the shred tile cannot verify, and is dropped unmatched if its shred or FEC set is already held.
The deadline (about 1 s after req_time) passed with no answer. Counts toward banning the peer only if no message of the same pick was answered, see ban.
The deadline passed with no answer to a highest_window_index or orphan message, or to a message for a slot at or below the root, including one rooted while in flight; never bans.
A metadata reply from the peer's address failed its proof; the peer is banned for 10 s.
Dropped unsettled because the pending request table was full; ours, not the peer's; never bans.
A highest_window_index or orphan message with no answer whose answer arrived another way before the deadline: the block's length, or its parent's ancestry, became known; a reply may have been a shred already held, dropped unmatched; never bans.
How the peer was picked.
Used in:
First attempt: the fastest available peer, an unmeasured one every 16th pick.
A later attempt, after the last peer's hedge time, preferring any peer but the last one asked.
A later attempt for a block not known only from turbine, from staked peers only.
A later attempt after 3 in a row brought no new shred of the FEC set: a random peer among the top 100 stake ranks, higher ranks likelier.
Role of the block asked about in its slot when sent.
Used in:
Neither the slot's notarized nor finalized block: turbine's block, or another block of the slot known by id.
The slot's first notarized, or stronger, block.
The slot's finalized block.
Kind of request.
Used in:
Shreds of one FEC set, by slot and index.
The highest shred a peer holds of a slot, to learn the block's length.
The ancestry of a slot whose parent block is not held.
Alpenglow: a block's parent and FEC set count, with a proof against its block id.
Alpenglow: the merkle root of one FEC set, with a proof against its block id.
Alpenglow: shreds of one FEC set of a specific block.
One row per block emitted with all the block-level diffs and results
Used in:
Monotonic sequence number identifying this block within the current run; the join key from runtime_txn / slot_confirmed / block_equivocated. Restarts at 1 each time a snapshot is loaded, so pair it with the stream's boot id. 0 means unavailable.
Block slot number
Block id of the slot. Under Tower this is the merkle root of the last FEC set (the last fec_merkle_roots entry). Under Alpenglow it is the block's double merkle root (DMR), which is not any FEC set's merkle root
Slot of the parent block
Parent slot's block id
Epoch this block belongs to
Block height (count of ancestor blocks) of this block in the chain
Scheduled leader identity pubkey for this block
Bank hash of this slot
Bank hash of the parent slot
Accounts lattice hash after this slot
Proof-of-history hash at the end of the slot (same as blockhash)
Decoded Clock sysvar for this slot
Newest 151 entries of the blockhash queue at the end of this slot, newest first: exactly the blockhashes a transaction in a child block may reference (age <= 150 inclusive, matching the agave off-by-one in the transaction age check). The first entry is this block's blockhash.
Decoded RecentBlockhashes sysvar account content at the end of this slot, newest first (the newest 150 blockhash queue entries with their fee calculators)
Highest hard fork slot less than or equal to bank slot
Total number of transactions committed in this block
Number of transactions that committed with an error
Number of non-vote transactions committed in this block
Number of non-vote transactions that committed with an error
Total number of signatures processed in the block
Number of data shreds making up this block
Cumulative tick height at the end of this block
Total execution (base) fees collected in this block, pre-burn
Total priority fees collected in this block (priority fees are not burned)
Total gross tips credited to tip accounts in this block, before block-level tip commission
Lamports burned at fee settlement per the fee schedule (50% of execution fees). If the resolved fee collector fails validation the reward portion is also burned; that case shows up as leader_fee_reward with no matching other_diffs credit.
Lamports owed for block production at fee settlement per the fee schedule (priority fees plus unburned execution fees). The actual credit is an other_diffs entry for the resolved fee collector: the leader identity, or the leader vote account's block-revenue collector when custom_commission_collector is active. If the collector fails validation this amount is burned instead and the collector's other_diffs entry shows no balance change.
Total lamport supply
Effective stake from stake delegations
Stake activating during this slot's epoch
Stake deactivating during this slot's epoch
Total stake delegated to active vote accounts this epoch
Final per-account diff of every sysvar account written this block; repeated writes to the same sysvar (e.g. the stake history update then its rent-exemption top-up at an epoch boundary) are coalesced into one entry keeping the first pre-state and last post-state. Ordinary slots carry ~4 entries (Clock, SlotHashes, RecentBlockhashes, SlotHistory; LastRestartSlot only when it changes), boundary/PER slots 5-6. Overflow past the 16-account bound drops the entry with a validator log warning
Final per-account diff of non-txn account updates that are not sysvar writes or epoch reward credits (both vote and stake epoch-reward credits are excluded here; see the runtime_reward table). Includes: the block fee credit at settlement (pubkey is the resolved fee collector: the leader identity unless custom_commission_collector routes it to the leader vote account's block-revenue collector; a zero-delta entry means collector validation failed and the reward burned), feature accounts at activation, core-BPF migration writes, and builtin program provisioning. Repeated writes to one account coalesce as in sysvar_diffs. Deliberately excluded as unbounded per-account floods: the per-account VAT-burn debits (see runtime_epoch.vat_burn_per_vote_account) and the per-block alpenglow cert-signer vote state rewrites. Steady state is one entry per block; 32 is an operational bound, not protocol-derived; overflow drops the entry with a validator log warning
Total number of FEC sets observed while assembling this block (not capped by fec_merkle_roots)
Merkle roots of the FEC sets observed while assembling this block, in order. Truncated to the first 512 roots — exact for fixed-size FECs (a block has at most 512), a prefix for legacy variable-size FECs when fec_count exceeds 512. Under Tower the last entry equals block_id; under Alpenglow block_id is the separate DMR and is not present in this list
Decoded Clock sysvar for this slot
Used in:
Clock sysvar slot (same as this row's slot)
Unix timestamp of the first slot of this epoch
Clock sysvar epoch
Epoch whose leader schedule is currently being computed
Stake-weighted cluster unix timestamp estimate for this slot
Post-state of an updated account
Used in:
Account pubkey
Post-state owner pubkey
Pre-state owner pubkey
Post-state lamports
Pre-state lamports
Post-state data size in bytes
Pre-state data size in bytes
True if the post-state account is executable
RecentBlockhashes sysvar entry
Used in:
Blockhash
Fee calculator (lamports per signature) in effect when this blockhash was registered
Post-state of an updated sysvar account
Used in:
Sysvar account pubkey
Post-state owner pubkey
Pre-state owner pubkey
Post-state lamports
Pre-state lamports
Post-state data size in bytes
Pre-state data size in bytes
True if the post-state account is executable
One row per epoch boundary with all the epoch-level diffs and results
Used in:
Monotonic sequence number identifying this block within the current run; the join key to runtime_block. Restarts at 1 each time a snapshot is loaded, so pair it with the stream's boot id. 0 means unavailable.
First replayed slot of the epoch-boundary bank (the bank that crosses into this epoch); with skipped slots this is not necessarily the epoch's nominal first slot
Parent slot on this fork (the last replayed slot of the previous epoch)
The new epoch being entered; the boundary snapshot captures stakes as of the end of epoch - 1
Effective stake across all delegations, recomputed with warmup/cooldown math at this boundary
Stake activating during the new epoch, recomputed at this boundary
Stake deactivating during the new epoch, recomputed at this boundary
Total stake delegated to the valid vote accounts snapshotted at this boundary
Vote accounts seen while accumulating delegations at this boundary
Vote accounts in the final boundary snapshot
Vote accounts that passed the validator-admission-ticket filters and were offered to the top-votes set
Smallest stake in the final boundary snapshot - the effective admission threshold
Decoded StakeHistory sysvar entry appended at this boundary
Decoded EpochSchedule sysvar
Feature ids newly activated at this boundary. The 16-entry bound is operational, not protocol-derived (no boundary on a real cluster has come close); overflow drops the entry with a validator log warning
Decoded EpochRewards sysvar as initialized at this boundary
Lamports burned from every admitted vote account at this boundary (SIMD-0357 VAT; alpenglow only, 0 otherwise). The burned set is exactly this boundary's runtime_vote_account rows, so the aggregate burned is this value times that row count. The debits are not emitted as account diffs, and the incinerator credit is not observable in runtime_block.other_diffs either: the incinerator is drained to zero later in the same block by fd_runtime_freeze, so its coalesced diff is 0 -> 0
Decoded EpochRewards sysvar as initialized at this boundary
Used in:
Total lamports minted for this epoch's rewards (vote + stake)
Lamports already distributed when the sysvar was initialized
Block height at which the stake rewards distribution starts
Number of stake reward partitions
Total reward points (sum of credits earned times stake across all delegations)
Parent blockhash seeding the partition hasher
Decoded EpochSchedule sysvar
Used in:
Slots per epoch
Slots before an epoch's start at which its leader schedule is computed
True if epochs ramp up from a shorter initial length, false otherwise
First epoch of full slots per epoch length
First slot of first normal epoch
Decoded StakeHistory sysvar entry appended at this boundary
Used in:
Epoch that just completed
Effective stake recorded for the completed epoch
Activating stake recorded for the completed epoch
Deactivating stake recorded for the completed epoch
One row per epoch-reward credit to an account
Used in:
Monotonic sequence number identifying this block within the current run; the join key to runtime_block. Restarts at 1 each time a snapshot is loaded, so pair it with the stream's boot id. 0 means unavailable.
Slot in which the credit was applied
Epoch the slot belongs to
Type of reward credit
Credited account pubkey: the stake account, the vote account, or the vote account's inflation collector (see kind)
Owner program of the credited account
Lamports before the credit
Lamports after the credit
Partitioned-rewards partition index this credit belongs to (0 for vote rewards)
New credits_observed stored on the stake account (0 for vote rewards)
Post-payout delegated stake written to the stake-delegations cache (0 for vote rewards)
Type of reward credit
Used in:
Vote commission credit in the first replayed bank that crosses the epoch boundary (not necessarily the epoch's nominal first slot when slots were skipped). Credited to the vote account itself, or, with custom_commission_collector active, to its inflation collector (possibly system-owned; one row per collector, rewards aggregated across the vote accounts routing to it)
Stake-account credit during the partitioned-epoch-rewards window
One row per root advance with the post-apply cache invariants
Used in:
Monotonic sequence number identifying this block within the current run; the join key to runtime_block. Restarts at 1 each time a snapshot is loaded, so pair it with the stream's boot id. 0 means unavailable.
Slot of the newly rooted block
Epoch the rooted slot belongs to
Slot of the previous root
Number of stake-delegations delta entries (non-tombstone) applied into the root map by this root advance. The advance covers every bank on (prev_root_slot, slot]. Each bank keeps at most one delta entry per stake account (last update wins), so this can be lower than the count of runtime_stake_delegation rows with kind upsert or reward whose slot is in that range when one account is mutated multiple times in a block.
Number of root stake-delegations removed by this root advance: tombstone delta entries (counted per account per bank, tombstones for accounts absent from the root map still count) plus any epoch-boundary inactive-stake prunes. Matches the count of runtime_stake_delegation rows with kind remove whose slot is in (prev_root_slot, slot] except when one account is removed more than once in a block.
Total entries in the root stake-delegations cache after this advance
Root stake-delegations effective-stake total after this advance
Root stake-delegations activating-stake total after this advance
Root stake-delegations deactivating-stake total after this advance
Block id (final FEC merkle root) of the newly rooted block
The complete history of the stake-delegations cache, one row per mutation: the baseline entries emitted once at boot (kind = bootup, one row per delegation in the finalized root cache after snapshot verification/refresh or genesis load, not the raw snapshot account stream), the entry rewrites applied by epoch-reward payouts (kind = reward), and the transaction-driven mutations emitted as the cache is updated at txn commit (kind = upsert / remove).
Used in:
Monotonic sequence number identifying this block within the current run; the join key to runtime_block. Restarts at 1 each time a snapshot is loaded, so pair it with the stream's boot id. 0 means unavailable.
Slot in which the mutating transaction was committed (the snapshot slot for bootup baseline entries; 0 for genesis-boot baselines)
Epoch the slot belongs to
0-indexed position of the mutating transaction within its block. UInt64 max for the non-transaction rows (bootup or reward kind)
First signature of the mutating transaction (64 bytes; zero for the non-transaction rows, bootup or reward kind)
Type of the cache entry mutation
Stake account pubkey
Vote account pubkey
Delegated lamports written to the cache
Delegation activation epoch written to the cache
Delegation deactivation epoch written to the cache
Vote credits_observed written to the cache
Type of the cache entry mutation
Used in:
Entry fully rewritten from the post-txn account state
Entry removed, either by a transaction (has signature and index_in_slot) or by epoch-boundary inactive-stake pruning (non-transaction; index_in_slot is UInt64 max)
Post-reconciliation baseline entry: emitted once at boot from the finalized root delegation cache (after snapshot verification/refresh, or genesis load), not from the raw snapshot account stream
Entry fully rewritten by an epoch-reward payout during the partitioned-rewards window
One row per dispatched transaction (committable or not), with the execution results
Used in:
Bank sequence number, counting from 1 within a run; the join key to block_completed. Not meaningful across restarts: scope by the stream's instance id from the connection's authentication metadata. 0 if unavailable.
Bank slot in which this transaction was executed
0-indexed position of this transaction within its block
0-indexed commit-completion order within this slot
FEC merkle root that was the current block_id of the bank when this txn was dispatched. All-zero on the leader path, which executes without FEC context.
Epoch the transaction was executed in
First signature of the transaction (64 bytes)
Blockhash referenced by the transaction
Fee-payer pubkey
True if the transaction was classified as a simple vote, false otherwise
True if the transaction is part of a bundle, false otherwise
True if the transaction was committable (landed on-chain)
True if the transaction landed but only fees were charged (e.g. account loading failure)
Transaction execution error
Instruction execution error; success unless txn_err = instruction_error
Source kind of exec_err; none unless txn_err = instruction_error
Instruction index that failed (UINT_MAX if not applicable)
Custom error code returned by the program (UINT_MAX if not applicable)
Compute unit limit requested by the transaction
Compute unit price (micro-lamports per CU) requested by the transaction
compute_unit_limit minus the remaining compute meter at commit
Requested heap size for the VM
Number of builtin program instructions in the transaction
Number of BPF (non-builtin) program instructions
Total bytes of account data loaded for this transaction
Configured loaded-accounts-data-size limit for this transaction
Net delta in writable account data sizes during execution (negative when the writable account data net shrunk)
Execution (base) fee charged to the fee payer, pre-burn (50% is burned at settlement)
Priority fee charged to the fee payer, pre-burn (priority fees are not burned)
Gross tips credited to tip accounts by this transaction, before the block-level tip commission
Number of signatures in the transaction
Cost-tracker signature cost
Cost-tracker write-lock cost
Cost-tracker data-bytes cost
Cost-tracker programs-execution cost
Cost-tracker loaded-accounts-data-size cost
Allocated accounts data size from the cost tracker
Per-account diffs for writable accounts that were modified
Every account the transaction locked as writable
Every account the transaction locked as readonly, in account-order
Distinct program pubkeys invoked at the top level of this transaction
Per-account diff entry
Used in:
Account pubkey
Post-state owner pubkey
Pre-state owner pubkey
Post-state lamports
Pre-state lamports
Post-state data size in bytes
Pre-state data size in bytes
True if the post-state account is executable
True if this write touched the stake cache
True if this write touched the vote cache
True if this write created a new vote account
True if this write removed a vote account
Instruction execution error; success unless txn_err = instruction_error
Used in:
instruction succeeded
The program instruction returned an error
The arguments provided to a program were invalid
An instruction's data contents were invalid
An account's data contents was invalid
An account's data was too small
An account's balance was too small to complete the instruction
The account did not have the expected program id
A signature was required but not found
An initialize instruction was sent to an account that has already been initialized.
An attempt to operate on an account that hasn't been initialized.
Program's instruction lamport balance does not equal the balance after the instruction
Program illegally modified an account's program id
Program spent the lamports of an account that doesn't belong to it
Program modified the data of an account that doesn't belong to it
Read-only account's lamports modified
Read-only account's data was modified
Executable bit on account changed, but shouldn't have
The instruction expected additional account keys
Program other than the account's owner changed the size of the account data
Failed to borrow a reference to account data, already borrowed
Account data has an outstanding reference after a program's execution
Allows on-chain programs to implement program-specific error types and see them returned by the runtime.
The return value from the program was invalid.
Executable accounts must be rent exempt
Unsupported program id
Cross-program invocation call depth too deep
An account required by the instruction is missing
Cross-program invocation reentrancy not allowed for this instruction
Length of the seed is too long for address generation
Provided seeds do not result in a valid address
Failed to reallocate account data of this length
Computational budget exceeded
Cross-program invocation with unauthorized signer or writable account
Failed to create program execution environment
Program failed to complete
Account is immutable
Incorrect authority provided
Failed to serialize or deserialize account data
An account does not have enough lamports to be rent-exempt
Invalid account owner
Program arithmetic overflowed
Unsupported sysvar
Provided owner is not allowed
Account data allocation exceeded the maximum accounts data size limit
Max accounts exceeded
Max instruction trace length exceeded
Builtin programs must consume compute units
Source kind of exec_err; none unless txn_err = instruction_error
Used in:
no instruction error
eBPF VM error
Syscall error
Instruction error
Transaction execution error
Used in:
Transaction succeeded
A Pubkey appears twice in the transaction's account_keys
Attempt to debit an account but found no record of a prior credit
Attempt to load a program that does not exist
Fee payer has insufficient lamports to cover the transaction fee
This account may not be used to pay transaction fees
The bank has seen this transaction before
The bank has not seen the given recent_blockhash or the transaction is too old and the recent_blockhash has been discarded.
An error occurred while processing an instruction, look at exec_err for more details
Transaction did not pass signature verification
This program may not be used for executing instructions
Transaction failed to sanitize accounts offsets correctly
Transaction would exceed max Block Cost Limit
Transaction would exceed max account limit within the block
Transaction would exceed account data limit within the block
Transaction locked too many accounts
Transaction loads an address table account that doesn't exist
Transaction loads an address table account with an invalid owner
Transaction loads an address table account with invalid data
Transaction address table lookup uses an invalid index
Transaction would exceed max Vote Cost Limit
Transaction would exceed total account data limit
Transaction contains a duplicate instruction that is not allowed
Transaction results in an account with insufficient funds for rent
Transaction exceeded max loaded accounts data size cap
Transaction requested a zero loaded-accounts-data-size limit
Sum of account balances before and after transaction do not match
Transaction was part of a bundle in which another transaction failed
Durable-nonce txn was rejected because its nonce account has already been advanced
Durable-nonce txn failed to advance its nonce account
Durable-nonce txn's recent_blockhash did not match the stored nonce
Transaction uses a message version that is not supported
One row per vote account per epoch boundary: the per-account vote-cache snapshot taken as the boundary is crossed
Used in:
Monotonic sequence number identifying this block within the current run; the join key to runtime_block. Restarts at 1 each time a snapshot is loaded, so pair it with the stream's boot id. 0 means unavailable.
First replayed slot of the new epoch
The new epoch being entered; the snapshot captures this account's stake as of the end of epoch - 1
Vote account
Validator identity pubkey
Effective stake delegated to this vote account at this boundary
Commission read from the vote account at this boundary (basis points)
True if the account existed in the snapshot taken one boundary earlier
Commission recorded one boundary earlier (0 unless has_commission_t_2)
True if the account existed in the snapshot taken two boundaries earlier
Commission recorded two boundaries earlier (0 unless has_commission_t_3)
Commission the reward calculation will actually use for this epoch's vote rewards: the oldest available of t-3 / t-2 / current when delay_commission_updates is active, current otherwise
Cumulative vote credits at this boundary
Cumulative vote credits at the start of the latest epoch-credits ladder entry
Entries in the vote account's epoch-credits ladder at this boundary
The validator received a valid shred
Used in:
The source IP address of the shred
The source port of the shred
The protocol used to send the shred
The raw shred payload bytes
The protocol used to send the shred
Used in:
Turbine
Repair
Leader
The validator generated a vote transaction
Used in:
Serialized signed vote transaction
The public key of the vote account
The public key that authorized the vote
The public key that pays for transaction fees
Signature identifying the transaction
Slot being voted on (top of tower)
Bank hash of the slot being voted on
Block ID of the slot being voted on
Recent blockhash parameter of the vote transaction
Tower of slots being voted on (oldest to newest)
Slot that was voted on
Used in:
Slot being voted on / locked out for
Confirmation count (higher = closer to root)
A block reached a commitment level. One row per block per level: a level crossed while the block is not tracked by local consensus (not yet replayed, or already discarded) reports once with forward true and bank_seq 0 and is not re-reported. Join to block_completed on bank_seq, or on slot and block_id (the only keys on forward rows).
Used in:
Bank sequence number, counting from 1 within a run; the join key to block_completed. Not meaningful across restarts: scope by the stream's instance id from the connection's authentication metadata (the boot id identifies the host kernel boot and survives validator restarts). 0 on forward rows. The boot bank (bank_seq 1) emits confirmations but no block_completed row. Forward confirmations for blocks never completely received join to no row on any key.
Slot number of the block.
Block identifier; with slot, the alternate join key to block_completed.
Stake in lamports voted for this exact block when the row was emitted. Only the row that triggered a propagated/duplicate/optimistic/super confirmation crossed the level's threshold; ancestors confirmed transitively report their own smaller tally. processed rows carry 0: no replay votes exist yet at the block's own completion. rooted rows carry replay-observed vote stake for the block plus descendants, no threshold. 0 if unavailable.
Total cluster stake in lamports for the epoch the tally was made in (captured at replay completion for processed and rooted rows). 0 if unavailable.
Whether the block is eligible for fork choice. An equivocating block becomes eligible again once duplicate confirmed. Always true on forward and ignored rows; validity is not assessed for blocks consensus is not tracking.
Commitment level reached, weakest to strongest: processed, propagated, duplicate, optimistic, super, rooted. ignored is a terminal outcome, not a level. A block never named directly by any vote emits no propagated/duplicate/optimistic/super rows even when confirmed transitively through a descendant, and the transitive walk stops at it, so voted ancestors above it can miss those rows too; all still emit processed and rooted rows.
Whether the level was crossed by cluster votes (gossip, TPU, or replayed blocks) while the block was not tracked by local consensus: not yet replayed, or already discarded (pruned losing fork, or completed after the root passed it). Always false for processed, ignored, and rooted rows, which local replay or rooting triggers.
Commitment level reached, weakest to strongest: processed, propagated, duplicate, optimistic, super, rooted. ignored is a terminal outcome, not a level. A block never named directly by any vote emits no propagated/duplicate/optimistic/super rows even when confirmed transitively through a descendant, and the transitive walk stops at it, so voted ancestors above it can miss those rows too; all still emit processed and rooted rows.
Used in:
Replay finished a block consensus had already pruned, so no processed row is emitted. Forward vote-level rows for the block may precede this.
Replayed locally, on any fork; no cluster stake required. Solana's 'processed' commitment.
More than a third of stake observed the block or a descendant.
More than 52% of stake voted for this exact block or a descendant; the cluster converges on it even if the slot equivocates.
More than two thirds of stake voted for the block or a descendant. Solana's 'confirmed' commitment.
More than four fifths of stake voted for the block or a descendant. Used when waiting for a supermajority at boot.
The block or a descendant reached maximum lockout and cannot be rolled back. The strongest level; Solana's 'finalized' commitment.
The result of a snapshot production attempt.
Used in:
Snapshot production result.
The snapshot slot. The snapshot contains the chain state after replaying this slot.
For an incremental snapshot, the slot of the full snapshot it's based on. 0 for a full snapshot.
Final snapshot filename.
Raw 32-byte snapshot hash (currently blake3_256(accounts_lthash)).
Bank hash of the snapshot slot.
Block identifier (FEC set tree root) of the bank at the snapshot slot.
Raw 2048-byte accounts LtHash from the bank.
Epoch number of snapshot slot.
Block height at the snapshot slot (as seen in manifest).
Total capitalization in lamports (as seen in manifest).
Cumulative number of historical transactions (as seen in manifest, unverified).
Number of account records written to the snapshot. Full snapshots contain all live accounts; incremental snapshots contain accounts changed since base_slot.
Number of zero-lamport account records written to the snapshot.
Number of non-tombstone account records read from the accounts database cache.
Number of non-tombstone account records read from accounts database disk storage.
Uncompressed serialized manifest payload size in bytes, excluding its TAR header and padding.
Total uncompressed size in bytes of account records, including each record header and record alignment padding but excluding TAR headers and TAR padding.
Uncompressed serialized status-cache payload size in bytes, excluding its TAR header and padding.
Final compressed snapshot file size in bytes, including archive data and skippable alignment frames.
Total number of bytes produced by decompressing all Zstandard frames, including TAR headers and TAR padding but excluding skippable alignment frames.
Number of Zstandard data frames in the snapshot file. Excludes skippable frames used for alignment padding.
Total compressed-file bytes occupied by Zstandard skippable frames used for alignment padding, including their 8-byte frame headers.
Zstandard compression window size in bytes.
Zstandard compression strategy.
Aggregate CPU time spent in Zstandard compression calls across snapshot producer threads, in nanoseconds.
Aggregate time snapshot producer threads spent blocked on snapshot file writes, in nanoseconds.
Wall-clock time at which snapshot production began, in nanoseconds since the Unix epoch.
Wall-clock time at which writing snapshot account records began, in nanoseconds since the Unix epoch.
Wall-clock time at which snapshot production ended, in nanoseconds since the Unix epoch.
Snapshot production result.
Used in:
The snapshot was produced and published to the local snapshot directory.
The incremental snapshot could not be produced because more accounts changed than snapshots.max_incremental_snapshot_accounts permits.
Zstandard compression strategy.
Used in:
Zstandard fast strategy.
Zstandard double-fast strategy.
Zstandard greedy strategy.
Zstandard lazy strategy.
Zstandard lazy2 strategy.
Zstandard binary-tree lazy2 strategy.
Zstandard binary-tree optimal strategy.
Zstandard binary-tree ultra strategy.
Zstandard binary-tree ultra2 strategy.
The validator received a verified, deduplicated transaction
Used in:
The source IP address of the transaction sender
The source port of the transaction sender
The protocol used to submit the transaction (e.g., QUIC, UDP, ...)
The ID of the bundle this transaction belongs to, if any. Zero indicates not a bundle transaction
The total number of transactions in the bundle this transaction belongs to, if any
The commission percentage for the bundle this transaction belongs to, if any
The public key of the commission recipient for the bundle this transaction belongs to, if any
The raw transaction payload bytes
The protocol used to submit the transaction (e.g., QUIC, UDP, ...)
Used in:
QUIC
UDP
Gossip
Bundle
Internal vote sender