package pulse.config.processor.v1

Mouse Melon logoGet desktop application:
View/edit binary Protocol Buffers messages

message AggregationConfig

aggregation.proto:16

Configuration for the aggregation processor.

Used in: ProcessorConfig

message AggregationConfig.Counters

aggregation.proto:60

Used in: AggregationConfig

message AggregationConfig.Counters.AbsoluteCounters

aggregation.proto:61

Used in: Counters

message AggregationConfig.Counters.Extended

aggregation.proto:73

Extended output configuration for counters.

Used in: Counters

message AggregationConfig.Gauges

aggregation.proto:96

Used in: AggregationConfig

message AggregationConfig.Gauges.Extended

aggregation.proto:98

Extended output configuration for gauges.

Used in: Gauges

message AggregationConfig.Histograms

aggregation.proto:120

Used in: AggregationConfig

message AggregationConfig.QuantileTimers

aggregation.proto:17

Used in: AggregationConfig

message AggregationConfig.QuantileTimers.Extended

aggregation.proto:19

Extended output configuration for timers.

Used in: QuantileTimers

message AggregationConfig.ReservoirTimers

aggregation.proto:45

Used in: AggregationConfig

message AggregationConfig.Summaries

aggregation.proto:125

Used in: AggregationConfig

message BufferConfig

buffer.proto:17

This is a basic buffer processor that accepts incoming metrics up to a limit before dropping. A fixed number of consumers pull metric batches and move them further along the pipeline. In the future we can add more sophisticated functionality such as spill to disk, a remote queue, etc.

Used in: ProcessorConfig

message CardinalityLimiterConfig

cardinality_limiter.proto:19

Configuration for a cardinality limiting processor. The processor uses Cuckoo filters to keep track of each metric ID without using very much space. Multiple buckets can be configured so that the limits smoothly roll across multiple time periods. When checking for the limit, only the first bucket is consulted. After this check, the metric is added to all buckets which provides the smooth rotation.

Used in: ProcessorConfig

message CardinalityLimiterConfig.GlobalLimit

cardinality_limiter.proto:21

Global limit configuration.

Used in: CardinalityLimiterConfig

message CardinalityLimiterConfig.PerPodLimit

cardinality_limiter.proto:27

Per-pod limit configuration.

Used in: CardinalityLimiterConfig

message CardinalityTrackerConfig

cardinality_tracker.proto:17

Track cardinality of different metric groupings and output them on a built-in /cardinality_tracker admin endpoint. In the future this information will be sent via API to the SaaS for display. We can also considering emitting metrics.

Used in: ProcessorConfig

message CardinalityTrackerConfig.Count

cardinality_tracker.proto:21

Approximates the cardinality of a specific metric name, matched based on regular expression. Every metric that matches will be counted. This uses a HyperLogLog algorithm to approximate the cardinality so it is lightweight from a memory perspective.

Used in: TrackingType

message CardinalityTrackerConfig.TopK

cardinality_tracker.proto:38

Approximates the top K cardinality of a specific metric grouping. The way this works is as follows: 1. In the first phase of each counting interval, a streaming topk algorithm (filtered space saving) is used to track the relative frequency of each grouping, as determined by the number of increments attributed to the grouping. (The grouping will be incremented for every metric that matches the grouping, multiple times for different tag sets.) 2. Once the top K groupings are known, the second phase of counting will maintain a hash map of exact grouping to HyperLogLog sketches. This will acquire a roughly accurate total cardinality count for each top grouping. The purpose of this is to avoid having to keep a first tier HashMap of all possible groupings which would be very memory intensive. Both phases are repeated over time to maintain a roughly accurate accounting with limited memory usage.

Used in: TrackingType

message CardinalityTrackerConfig.TopK.TagList

cardinality_tracker.proto:39

Used in: TopK

message CardinalityTrackerConfig.TrackingType

cardinality_tracker.proto:60

Used in: CardinalityTrackerConfig

message DropConfig

drop.proto:137

Used in: DropProcessorConfig, drop_tester.v1.DropTestCase

message DropProcessorConfig

drop.proto:144

Used in: ProcessorConfig

message DropRule

drop.proto:14

Used in: DropConfig

message DropRule.AndMatch

drop.proto:15

Used in: DropCondition

message DropRule.DropCondition

drop.proto:87

Used in: DropRule, AndMatch

enum DropRule.DropMode

drop.proto:112

Used in: DropRule

message DropRule.SimpleValueMatch

drop.proto:59

Used in: ValueMatch

message DropRule.StringMatch

drop.proto:19

Used in: DropCondition, TagMatch

message DropRule.TagMatch

drop.proto:31

Used in: DropCondition

message DropRule.TimestampAgeMatch

drop.proto:80

Used in: DropCondition

message DropRule.ValueMatch

drop.proto:67

Used in: DropCondition

enum DropRule.ValueMatchOperator

drop.proto:39

Used in: SimpleValueMatch

message ElisionConfig

elision.proto:16

Configuration for the elision processor.

Used in: ProcessorConfig

message ElisionConfig.BlocklistConfig

elision.proto:87

Blocklist configuration.

Used in: ElisionConfig

message ElisionConfig.EmitConfig

elision.proto:18

Emission configuration.

Used in: ElisionConfig, RegexOverrideConfig

message ElisionConfig.EmitConfig.ConsistentEveryPeriod

elision.proto:19

Used in: EmitConfig

message ElisionConfig.RegexOverrideConfig

elision.proto:114

Configures overrides for emission based on metric name regex.

Used in: ElisionConfig

message ElisionConfig.ZeroElisionConfig

elision.proto:47

Zero elision configuration.

Used in: ElisionConfig

message ElisionConfig.ZeroElisionConfig.Counters

elision.proto:49

Configuration for counters.

Used in: ZeroElisionConfig

message ElisionConfig.ZeroElisionConfig.Counters.AbsoluteCounters

elision.proto:51

Configuration for absolute counters.

Used in: Counters

message ElisionConfig.ZeroElisionConfig.Histograms

elision.proto:66

Configuration for histograms.

Used in: ZeroElisionConfig

message InternodeConfig

internode.proto:21

Configuration for the internode processor. Internode performs consistent hashing of metrics and sends them to other nodes in the cluster as necessary. This insures that processors that depend on seeing the same metric consistently will see it. Currently the internode cluster configuration is hard coded and not dynamically discovered. This can be accomplished using Kubernetes stateful sets or some other similar mechanism that provides stable DNS for cluster members.

Used in: ProcessorConfig

message InternodeConfig.HealthCheckConfig

internode.proto:31

Used in: InternodeConfig

message InternodeConfig.NodeConfig

internode.proto:23

Configuration for an internode node.

Used in: InternodeConfig

message InternodeConfig.RequestPolicy

internode.proto:46

Configuration for internode requests.

Used in: InternodeConfig

message MutateConfig

mutate.proto:76

Processor that performs basic mutation based on metadata. Mutation is performed by a VRL program, which is documented at a high level here: https://vector.dev/docs/reference/vrl/. Pulse supports a limited set of operations on metrics, and provides context metadata that can be used during transforms. The following context is provided: .name is the name of the metric. Can be modified. .tags are the metric tags as a map. Can be modified. .mtype is the type of the metric. Cannot be modified. If using Kubernetes discovery, the following metadata is provided: %k8s.service.name is the name of the service if service discovery is enabled and there is an associated service. %k8s.namespace is the namespace of the pod. %k8s.pod.name is the name of the pod. %k8s.pod.ip is the IP of the pod. %k8s.pod.labels are the labels of the pod. %k8s.pod.annotations are the annotations of the pod. %k8s.node.name is the name of the node. %k8s.node.ip is the IP of the node. %k8s.node.labels are the labels of the node. %k8s.node.annotations are the annotations of the node. %prom.scrape.address is <IP>:<PORT> if the target has been scraped. Errors and abort() are handled differently. Errors will emit a periodic warnings and are assumed to be real operator errors. Aborts via abort() are assumed to be intentional with the desire to drop the metric. Early return is possible using the return statement. The expression passed to return will be ignored so it can be anything, such as "return true". It is possible to "flatten" Prometheus histograms and summaries into multiple component metrics. For histograms this emits multiple <metric>_bucket metrics, a <metric>_count metric, and a <metric>_sum metric. For summaries this emits multiple <metric> metrics with different quantiles and a <metric>_sum and <metric>_count metric. This is done by setting the following pseudo field to true: .flatten_prom_histogram_and_summary = true Pulse also implements some custom functions beyond what is available from core VRL. These include: - pulse_inc_counter(name: string, value: int, tag_names: array, tag_values: array): Increments a counter with the given name by the given value. The name must be a constant string. name: (required) The name of the counter to increment. Must be a constant string. value: (required) The value to increment the counter by. Must be an integer. tag_names: (optional) An array of tag names to add to the counter. tag_values: (optional) An array of tag values to add to the counter. Required if tag_names is provided. Must be the same length as tag_names. - pulse_log(value: any, level: string, rate_limit_secs: int): Outputs a rate limited log. value: (required) The value to log of any type level: (optional) The log level to use. Must be one of "trace", "debug", "info", "warn", "error". Defaults to "info". rate_limit_secs: (optional) The rate limit in seconds for the message. Defaults to 1 second. - pulse_add_to_downstream_id(suffix: bytes): Adds the given suffix to the metric's downstream ID. This can be used to create different downstream IDs based on metadata. The suffix is added after a colon (:) separator. If the downstream ID is not InflowProvided, it will be converted to InflowProvided first. This useful for aggregating metrics based on a particular metadata value before stripping it. For example, using the pod name before stripping it will allow proper gauge/counter aggregations for difference downstream ID sources while still producing a single output aggregated metric.

Used in: ProcessorConfig

message PopulateCacheConfig

populate_cache.proto:14

The configuration for the populate cache processor. Cache population is required before calling the aggregation, elision, or sampler processor. TODO(mattklein123): Verify the previous during pipeline setup.

Used in: ProcessorConfig

(message has no fields)

message ProcessorConfig

processor.proto:25

Configuration for an individual processor.

Used in: bootstrap.v1.PipelineConfig

message RegexConfig

regex.proto:16

Configuration for a regex allow/deny processor. Metrics can either be explicitly allowed (any that don't match will be dropped) or explicitly denied (any that do match will be dropped). Both types of rules can be supplied. While these rules can be expressed as VRL programs and aborts in the mutate filter, this processor is provided for performance reasons. The regular expression syntax is the one documented here: https://docs.rs/regex/latest/regex/#syntax

Used in: ProcessorConfig

message SamplerConfig

sampler.proto:16

The configuration for the sampler processor. This processor will cache metrics and emit them onward every configured interval of time.

Used in: ProcessorConfig