Partitioning Processor
contrib
Maintainers: @paulojmdias, @pjanotti
Source: opentelemetry-collector-contrib
Supported Telemetry
Overview
[!NOTE] This component was originally developed by @axw in axw/otel-partitioning-processor and donated to this repository. See #39199.A processor that partitions incoming telemetry batches using OTTL value expressions. Each incoming batch is split into one or more sub-batches whose items all evaluate to the same set of partition key values. Each sub-batch is forwarded to the next consumer concurrently with the evaluated key/value pairs appended to the outgoing request metadata. Use cases
- Kafka topic routing and message keying — partition by a resource attribute to control which topic and/or partition key each batch lands on, without the multi-resource routing bug present in the Kafka exporter.
- Multi-tenant pipelines — split a mixed-tenant batch so that each downstream exporter call sees only one tenant’s data.
- Metadata-aware batching — combine with a metadata-keyed batch sender (see otelcol#10825) so that all data for a given partition key is batched together before export.
Configuration
keys
Required. A map from partition key name to an OTTL value expression that evaluates to a string. Each expression is evaluated against every item in the incoming batch; items that produce the same tuple of values are collected into the same partition.
Key names are used as client.Metadata keys, which are case-insensitive and stored lower-cased; names that differ only in case are rejected.
Expressions must evaluate to string, or nil. In case of a nil result (e.g. a missing attribute), the item is partitioned with other items that also produce nil for that key, the key is omitted from the outgoing metadata, and any inbound metadata value for that key is removed (see below). This allows you to partition by optional attributes without losing those items entirely.
If an expression fails to evaluate or returns any other type, the whole batch is rejected with a permanent error, so that receivers do not retry it.
Supported OTTL contexts per signal
The following contexts are not supported and will produce an error
at startup:
spanevent(would duplicate spans with the same TraceID/SpanID across partitions, breaking downstream trace assemblers)profilesample(may be introduced in the future if needed)
Outgoing request metadata
For each partition, the processor calls the next consumer with a new context carrying the evaluated partition key/value pairs inclient.Metadata:
nil; inbound values never leak through.
Partition deliveries are started concurrently, one goroutine per partition, in the order in which each partition was first seen in the batch. Every partition is delivered even if another one fails; the errors of all failed deliveries are joined and returned. The inbound context is passed through without a derived cancellation, so downstream consumers that keep the context after returning are not affected.
When every item in a batch maps to the same partition, the batch is forwarded as-is without copying. When a batch is split, items are moved (not copied) into the partitions, so the processor declares that it mutates data; if the pipeline fans out to several consumers, the collector clones the batch before it reaches this processor.
Using the metadata downstream
The processor only adds the partition keys to the request metadata; another component has to read them. A few things to keep in mind when wiring that up:- Exporter batching: when an exporter batches with
sending_queue::batch, requests from different partitions are merged, and only the metadata keys listed insending_queue::batch::partition::metadata_keysare kept. List every partition key that a downstream component reads there, which also makes the exporter batch each partition separately. Without batching, the exporter’s in-memory queue keeps the request metadata. - Inbound metadata: receivers only need
include_metadata: truewhen a key expression reads inbound request metadata throughotelcol.client.metadata. The keys this processor adds do not need it. - High-cardinality keys: every partition is delivered concurrently, so a key with many distinct values in one batch results in as many concurrent downstream calls. Put an exporter
sending_queuebetween this processor and slow or synchronous backends.
Examples
Partition by a resource attribute
client.Metadata["tenant_id"] = ["<tenant value>"].
Partition by several keys
(tenant.id, severity_text) tuple are grouped together, and the outgoing metadata carries both keys.
Set a per-tenant HTTP header
Split mixed-tenant batches and send each tenant’s data with its ownX-Scope-OrgID header, for example to a multi-tenant backend. The exporter batches per tenant, and the headers_setter extension reads the tenant from the request metadata.
Route to Kafka topics and set the record key
Write each service’s spans to its own Kafka topic, keyed by trace ID, with the Kafka exporter reading the topic and key from the request metadata. When batching is enabled, the Kafka exporter requires both keys insending_queue::batch::partition::metadata_keys.
Partition by inbound request metadata
Use a header sent by the client, such as a gRPC or HTTP header, as a partition key.otelcol.client.metadata["<header>"] returns a []string; select a single element with [0] to satisfy the string constraint. The receiver must set include_metadata: true for inbound headers to be available.
Attributes
Configuration
Example Configuration
Last generated: 2026-11-09