otelconnect

Build GoDoc

connectrpc.com/otelconnect adds support for OpenTelemetry tracing and metrics collection to Connect servers and clients.

For more on Connect, OpenTelemetry, and otelconnect, see the Connect announcement blog post and the observability documentation on connectrpc.com.

An example

package main

import (
	"context"
	"fmt"
	"log"
	"net/http"

	"connectrpc.com/connect/v2"
	"connectrpc.com/connect/v2/connecthttp"
	"connectrpc.com/otelconnect"
	// Generated from your protobuf schema by protoc-gen-go and
	// protoc-gen-connect-go.
	pingv1 "connectrpc.com/otelconnect/internal/gen/observability/ping/v1"
	"connectrpc.com/otelconnect/internal/gen/observability/ping/v1/pingv1connect"
)

func main() {
	otelInterceptor, err := otelconnect.NewServerInterceptor()
	if err != nil {
		log.Fatal(err)
	}

	// otelconnect.NewServerInterceptor provides an interceptor that adds
	// tracing and metrics to handlers; otelconnect.NewClientInterceptor does
	// the same for clients. By default, they use OpenTelemetry's global
	// TracerProvider and MeterProvider, which you can configure by following
	// the OpenTelemetry documentation. If you'd prefer to avoid globals, use
	// otelconnect.WithTracerProvider and otelconnect.WithMeterProvider.
	server := connect.NewServer(otelInterceptor)
	pingv1connect.RegisterPingServiceHandler(server, &pingv1connect.UnimplementedPingServiceHandler{})

	mux := http.NewServeMux()
	connecthttp.Mount(mux, server)
	http.ListenAndServe("localhost:8080", mux)
}

func makeRequest() {
	otelInterceptor, err := otelconnect.NewClientInterceptor()
	if err != nil {
		log.Fatal(err)
	}

	client := pingv1connect.NewPingServiceClient(
		connect.NewClient(
			connecthttp.NewTransport(http.DefaultClient, "http://localhost:8080"),
			otelInterceptor,
		),
	)
	resp, err := client.Ping(context.Background(), &pingv1.PingRequest{})
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(resp)
}

Configuration for internal services

By default, instrumented servers are conservative and behave as though they're internet-facing. They don't trust any tracing information sent by the client, and will create new trace spans for each request. The new spans are linked to the remote span for reference (using OpenTelemetry's trace.Link), but tracing UIs will display the request as a new top-level transaction.

If your server is deployed as an internal service, configure otelconnect to trust the client's tracing information using otelconnect.WithTrustRemote. With this option, servers will create child spans for each request.

Semantic conventions

otelconnect follows the OpenTelemetry RPC semantic conventions (semconv v1.43.0). Spans and metrics carry rpc.system.name (connectrpc or grpc), the fully-qualified rpc.method, rpc.response.status_code and, on failure, error.type. Status codes are the uppercase Connect codes (NOT_FOUND, DEADLINE_EXCEEDED) for every RPC system, or OK. Client telemetry adds server.address and server.port; server spans can add network.peer.address and network.peer.port (see below).

Each side records one metric, rpc.{server,client}.call.duration, in seconds. WithDurationHistogramOptions changes its buckets, and a Labeler adds attributes to metrics from a handler or client.

Reducing tracing cardinality

The OpenTelemetry RPC conventions tag server spans with the remote client's address and ephemeral port. These are high-cardinality, so otelconnect omits them by default; opt back in with otelconnect.WithServerPeerAttributes. For more customizable attribute filtering, use otelconnect.WithAttributeFilter; to skip RPCs entirely, use otelconnect.WithFilter.

Interceptors run inside the Connect handler, so HTTP middleware cannot see their span. To trace at the HTTP layer too, wrap the mux with otelhttp; the interceptor finds its span in the request context and creates the RPC span as a child. Note that otelhttp then decides whether to trust incoming trace headers, not otelconnect.WithTrustRemote.

Status

Unary Streaming Client Streaming Handler
Metrics ✅ ✅ ✅
Tracing ✅ ✅ ✅

Ecosystem

Support and Versioning

otelconnect supports:

Legal

Offered under the Apache 2 license.