package pulse.config.inflow.v1

Get desktop application:
View/edit binary Protocol Buffers messages

Advanced configuration for socket based inflows.

Used in: TcpServerConfig, UnixServerConfig

• optional google.protobuf.Duration idle_timeout = 1

  How long to wait for a connection to receive data before it is shutdown for being idle. Defaults to 60s.

• optional uint64 buffer_size = 2

  How much data to buffer on an incoming socket before applying back pressure. Defaults to 8KiB.

• optional google.protobuf.Duration max_connection_duration = 3

  The maximum amount of time an incoming connection will be kept alive. Can be usefull to force connection rotation and fix imbalance for downstreams. Defaults to none.

Configuration for determining the downstream ID of incoming metrics.

Used in: OtlpServerConfig, PromRemoteWriteServerConfig

• oneof source_type

  • bool remote_ip = 1

    Uses the remote IP address of the incoming connection.

  • string request_header = 2

    Uses the value found in the specified request header. If the request header is not found, falls back to remote IP address.

• bool append_tags_to_downstream_id = 3

  Whether to append the incoming metric tags to the downstream ID. When doing aggregation using Prometheus absolute counters, the aggregation processor must know all unique sources of a metric so that absolute counter deltas can be correctly calculated. If tags are removed using a mutate processor, the aggregation processor will no longer be able to determine unique sources. By appending the original tags to the incoming ID, tags can be removed and then forwarded via internode and aggregated correctly. Depending on the aggregation topology this is likely required if absolute counters are being used.

Configuration for an individual inflow.

Used in: bootstrap.v1.PipelineConfig

• repeated string routes = 1

  The routes that the inflow sends to.

• oneof config_type

  • MetricGeneratorConfig metric_generator = 2

  • PromRemoteWriteServerConfig prom_remote_write = 3

  • TcpServerConfig tcp = 4

  • UdpServerConfig udp = 5

  • UnixServerConfig unix = 6

  • KubernetesPrometheusConfig k8s_prom = 7

  • OtlpServerConfig otlp = 8

Collect prometheus metrics by discovering endpoints via Kubernetes. The annotations checked are the standard Prometheus annotations such as prometheus.io/scrape, prometheus.io/path, etc.

Used in: InflowConfig

• optional google.protobuf.Duration scrape_interval = 1

  How often eligible Prometheus endpoints are scraped.

• oneof target

  • KubernetesPrometheusConfig.Endpoint endpoint = 2

  • KubernetesPrometheusConfig.Node node = 3

  • KubernetesPrometheusConfig.Pod pod = 4

  • KubernetesPrometheusConfig.HttpServiceDiscovery http_service_discovery = 8

  • KubernetesPrometheusConfig.RemoteEndpoint remote_endpoint = 9

• bool emit_up_metric = 5

  Whether to emit the synthetic "up" metric as per https://prometheus.io/docs/concepts/jobs_instances/. If true, for each scrape target "up" will be a gauge set to 0 or 1 depending on whether the scrape was successful. The "instance" tag will be set to the target's address. The "job" tag will not currently be set.

• optional KubernetesPrometheusConfig.TLS tls_config = 6

  Based on https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tls_config allow users to add cert and key files

• optional google.protobuf.Duration scrape_timeout = 7

  The timeout to use on the scrape request. If not set this defaults to 15s.

Scrapes metrics from Prometheus endpoints defined on pods via associated service annotations. Resolves to all pod-local endpoints using the service annotations to infer path, port, etc.

Used in: KubernetesPrometheusConfig

• repeated UseK8sHttpsServiceAuthMatcher use_k8s_https_service_auth_matchers = 2

  If any of the optionally provided matchers matches, HTTPS K8s service auth will be used against the scrape target.

An implementation of the Prometheus HTTP service discovery protocol as documented here: https://prometheus.io/docs/prometheus/latest/http_sd/

Used in: KubernetesPrometheusConfig

• optional common.v1.EnvOrInline url = 1

  The URL to scrape for service discovery. This should be a valid URL.

• optional google.protobuf.Duration fetch_interval = 2

  The interval to use for fetching the service discovery config. If not set default to 1 minute.

Used in: Pod, RemoteEndpoint

• oneof filter_type

  • string container_port_name_regex = 1

    Apply the following regex to each container port name to determine if it should be scraped. If the regex is a match, the container port will be scraped.

  • InclusionFilter.AnnotationMatcher annotation_matcher = 2

    This allows extracting a scrape port from a generic annotation. Optionally it allows matching on a different annotation for inclusion versus extracting the port.

Used in: InclusionFilter

• string inclusion_annotation_name = 1

  Which annotation to use for deciding whether to scrape.

• string inclusion_annotation_regex = 2

  The regex to match the name against.

• string port_annotation_name = 3

  The annotation to use for deciding which port to scrape. This can be the same as the inclusion annotation name if desired.

Scrape metrics from endpoints discovered on the node. This will resolve to the kubelet endpoint on the local node.

Used in: KubernetesPrometheusConfig

• string path = 1

  The path to the prometheus endpoint on the node.

Scrapes metrics from Prometheus endpoints defined on pods via pod annotations.

Used in: KubernetesPrometheusConfig

• repeated InclusionFilter inclusion_filters = 1

  Inclusion filters can be used to include pods to be scraped even if they do not have the standard Prometheus annotations. An inclusion filter if successful will result in a container port to be scraped.

• repeated UseK8sHttpsServiceAuthMatcher use_k8s_https_service_auth_matchers = 2

  If any of the optionally provided matchers matches, HTTPS K8s service auth will be used against the scrape target.

• optional PrometheusAnnotationOverrides annotation_overrides = 3

  Optional overrides for the base Prometheus annotations used to discover scrape targets.

Used in: Pod

• optional string scrape = 1

  If not set, defaults to "prometheus.io/scrape".

• optional string namespace = 2

  If not set, defaults to "prometheus.io/namespace".

• optional string path = 3

  If not set, defaults to "prometheus.io/path".

• optional string scheme = 4

  If not set, defaults to "prometheus.io/scheme".

• optional string port = 5

  If not set, defaults to "prometheus.io/port".

Watch a Kubernetes endpoints resource and scrape all ready endpoints remotely.

Used in: KubernetesPrometheusConfig

• string namespace = 1

  The namespace of the Kubernetes endpoints resource to watch.

• string service = 2

  The name of the Kubernetes endpoints resource to watch.

• repeated InclusionFilter inclusion_filters = 3

  Inclusion filters can be used to include endpoints to be scraped even if they do not have the standard Prometheus annotations. An inclusion filter if successful will result in a endpoint port to be scraped.

• repeated UseK8sHttpsServiceAuthMatcher use_k8s_https_service_auth_matchers = 4

  If any of the optionally provided matchers matches, HTTPS K8s service auth will be used against the scrape target.

Used in: KubernetesPrometheusConfig

• optional string cert_file = 1

• optional string key_file = 2

• bool insecure_skip_verify = 3

Used in: Endpoint, Pod, RemoteEndpoint

• oneof auth_matcher

  • UseK8sHttpsServiceAuthMatcher.KeyValue annotation_matcher = 1

    Matches if the supplied key/value pair is present in the pod's annotations.

  • bool always = 2

    Always matches.

Used in: UseK8sHttpsServiceAuthMatcher

• string key = 1

• optional string value = 2

  If not supplied, the value is ignored and only key presence is checked.

Configuration for the metric generator inflow.

Used in: InflowConfig

• optional common.v1.WireProtocol protocol = 1

  Which protocol to generate.

• optional uint64 n_tasks = 2

  The number of tasks to spawn for generating metrics. Defaults to 16.

• optional uint64 batch_size = 3

  How many metrics to emit per flush interval. This is divided between all tasks. Defaults to 128.

• optional google.protobuf.Duration flush_interval = 4

  How often to emit batches. Defauls to 1s.

Supports OTLP over HTTP+protobuf.

Used in: InflowConfig

• string bind = 1

  Which address to bind to.

• optional DownstreamIdSource downstream_id_source = 2

  How to determine the downstream ID of incoming metrics. If not specified, defaults to `remote_ip`.

• optional bool include_resource_attributes = 3

  If true (the default), resource attributes will be included as metric tags. Scope or data point attributes with the same name will override resource attributes.

• optional bool include_scope_attributes = 4

  If true (the default), scope attributes will be included as metric tags. Data point attributes with the same name will override scope attributes.

• optional string path = 5

  The HTTP path to listen on. If not specified, defaults to `/v1/metrics`.

Configuration for the Prometheus remote write server inflow.

Used in: InflowConfig

• string bind = 1

  Which address to bind to.

• optional PromRemoteWriteServerConfig.ParseConfig parse_config = 2

  Extra configuration for parsing incoming remote write protocol.

• optional DownstreamIdSource downstream_id_source = 3

  How to determine the downstream ID of incoming metrics. If not specified, defaults to `remote_ip`.

Parser settings for incoming remote write protocol.

Used in: PromRemoteWriteServerConfig

• bool summary_as_timer = 1

  https://github.com/lyft/statsrelay/ sets the Prometheus SUMMARY metric type for StatsD timer values. This is not compliant with the actual Prometheus data model.

• bool counter_as_delta = 2

  https://github.com/lyft/statsrelay/ sets the Prometheus COUNTER metric type for StatsD delta counter values. This is not compliant with the actual Prometheus data model.

• bool ignore_duplicate_metadata = 3

  https://github.com/lyft/statsrelay/ does not dedup metric family metadata in the metadata list. If set, ignore duplicate entries and do not warn about them.

• bool flatten_histogram_and_summary = 4

  If set, metadata for histogram and summary metrics will be ignored, and those metrics will be flattened with an unknown metric type. This is useful if the sender is splitting histograms and summaries across different requests. Note that this means that these metrics cannot be aggregated or have any other operations performed on them that depend on their type.

Configuration for a TCP based inflow.

Used in: InflowConfig

• string bind = 1

  Which address to bind to.

• optional common.v1.WireProtocol protocol = 2

  Which protocol is expected.

• optional AdvancedSocketServerConfig advanced = 3

  Advanced configuration for the inflow.

• bool bind_k8s_pod_metadata_by_remote_ip = 4

  If true, the KubernetesBootstrapConfig will be used to watch all pods on the node. Incoming metrics will be looked up by remote IP to match the origin pod and its metadata to the metrics.

• optional google.protobuf.Duration pre_buffer_window = 5

  An optional amount of time that metrics will be buffered by the inflow before being further dispatched. Practically, this is useful when relying on Kubernetes informer metadata and there is no external mechanism to synchronize pods fully starting (e.g., an init container) such that there is a potential race condition between pods sending statsd metrics prior to the informer being aware of the pod's existence. Defaults to disabled. 5s is a reasonable setting in most environments where the above conditions apply. Note that counters, gauges, and timers will be aggregated to avoid unbounded memory growth of the application sends a lot of metrics.

• bool always_pre_buffer = 6

  If this is true and a pre-buffer window is set, all incoming metrics will be pre-buffered continuously. This is useful if for example applications are sending larger numbers of timers and we would like to do as little processing as possible on initial ingestion.

• optional uint32 reservoir_size = 7

  The size of the reservoir for timers. If not set defaults to 100.

Configuration for a UDP based inflow.

Used in: InflowConfig

• string bind = 1

  Which address to bind to.

• optional common.v1.WireProtocol protocol = 2

  Which protocol is expected.

• bool bind_k8s_pod_metadata_by_remote_ip = 3

  If true, the KubernetesBootstrapConfig will be used to watch all pods on the node. Incoming metrics will be looked up by remote IP to match the origin pod and its metadata to the metrics.

• optional UdpServerConfig.PreBufferConfig pre_buffer = 4

  Configuration for UDP session pre-buffering. See above for more information. Defaults to disabled.

Used in: UdpServerConfig

• optional google.protobuf.Duration pre_buffer_window = 1

  An optional amount of time that metrics will be buffered by the inflow before being further dispatched. Practically, this is useful when relying on Kubernetes informer metadata and there is no external mechanism to synchronize pods fully starting (e.g., an init container) such that there is a potential race condition between pods sending statsd metrics prior to the informer being aware of the pod's existence. 5s is a reasonable setting in most environments where the above conditions apply. Note that counters, gauges, and timers will be aggregated to avoid unbounded memory growth of the application sends a lot of metrics.

• optional google.protobuf.Duration session_idle_timeout = 2

  Because UDP is a datagram protocol, there is no actual stream to key off of like there is for TCP. Every incoming IP is bound to a "session" which will timeout after the configured session idle timeout. This should be set as large as possible but less than any potential window for IP recycling. For example if pod IP addresses have a recycle cooldown of 30s, a reasonable setting for this might be 20s.

• bool always_pre_buffer = 3

  If this is true and a pre-buffer window is set, all incoming metrics will be pre-buffered continuously. This is useful if for example applications are sending larger numbers of timers and we would like to do as little processing as possible on initial ingestion.

• optional uint32 reservoir_size = 4

  The size of the reservoir for timers. If not set defaults to 100.

Configuration for a unix domain docket based inflow.

Used in: InflowConfig

• string path = 1

  The path to bind to.

• optional common.v1.WireProtocol protocol = 2

  Which protocol is expected.

• optional AdvancedSocketServerConfig advanced = 3

  Advanced configuration for the inflow.