> ## Documentation Index
> Fetch the complete documentation index at: https://otel.fyi/llms.txt
> Use this file to discover all available pages before exploring further.

# Partitioning

> OpenTelemetry processor for Partitioning

# Partitioning Processor

![Status](https://img.shields.io/badge/status-alpha-red)

**Available in:** `contrib`

**Maintainers:** [@paulojmdias](https://github.com/paulojmdias), [@pjanotti](https://github.com/pjanotti)

**Source:** [opentelemetry-collector-contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/partitioningprocessor)

## Supported Telemetry

![Logs](https://img.shields.io/badge/logs-alpha-blue) ![Metrics](https://img.shields.io/badge/metrics-alpha-green) ![Traces](https://img.shields.io/badge/traces-alpha-orange)

## Overview

> [!NOTE]
>
> This component was originally developed by [@axw](https://github.com/axw) in
> [axw/otel-partitioning-processor](https://github.com/axw/otel-partitioning-processor)
> and donated to this repository. See [#39199](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/39199).

A processor that partitions incoming telemetry batches using [OTTL](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/README.md) value expressions. Each incoming batch is split into one or more sub-batches whose items all evaluate to the same set of partition key values. Each sub-batch is forwarded to the next consumer concurrently with the evaluated key/value pairs appended to the outgoing [request metadata](https://pkg.go.dev/go.opentelemetry.io/collector/client#Metadata).

**Use cases**

* **Kafka topic routing and message keying** — partition by a resource attribute to control which topic and/or partition key each batch lands on, without the multi-resource routing bug present in the Kafka exporter.
* **Multi-tenant pipelines** — split a mixed-tenant batch so that each downstream exporter call sees only one tenant's data.
* **Metadata-aware batching** — combine with a metadata-keyed batch sender (see [otelcol#10825](https://github.com/open-telemetry/opentelemetry-collector/issues/10825)) so that all data for a given partition key is batched together before export.

## Configuration

```yaml theme={null}
processors:
  partitioning:
    keys:
      <key-name>: <OTTL value expression>
      ...
```

### `keys`

Required. A map from **partition key name** to an [OTTL value expression](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/README.md) that evaluates to a `string`. Each expression is evaluated against every item in the incoming batch; items that produce the same tuple of values are collected into the same partition.

Key names are used as `client.Metadata` keys, which are case-insensitive and stored lower-cased; names that differ only in case are rejected.

Expressions must evaluate to `string`, or `nil`. In case of a `nil` result (e.g. a missing attribute), the item is partitioned with other items that also produce `nil` for that key, the key is omitted from the outgoing metadata, and any inbound metadata value for that key is removed (see below). This allows you to partition by optional attributes without losing those items entirely.

If an expression fails to evaluate or returns any other type, the whole batch is rejected with a permanent error, so that receivers do not retry it.

### Supported OTTL contexts per signal

| Signal | Supported contexts |
| - | - |
| Logs | `otelcol`, `resource`, `scope`, `log` |
| Traces | `otelcol`, `resource`, `scope`, `span` |
| Metrics | `otelcol`, `resource`, `scope`, `metric`, `datapoint` |
| Profiles | `otelcol`, `resource`, `scope`, `profile` |

The following contexts are **not** supported and will produce an error
at startup:

* `spanevent` (would duplicate spans with the same TraceID/SpanID across partitions, breaking downstream trace assemblers)
* `profilesample` (may be introduced in the future if needed)

### Outgoing request metadata

For each partition, the processor calls the next consumer with a new context carrying the evaluated partition key/value pairs in [`client.Metadata`](https://pkg.go.dev/go.opentelemetry.io/collector/client#Metadata):

```
client.Metadata["<key-name>"] = ["<evaluated-value>"]
```

Any metadata already present on the inbound context is preserved, except for keys that share a name with a partition key: those are always removed from the inbound metadata, even when the expression produces no value for an item. The outgoing value for such a key is therefore always the partition value, or absent when the expression returns `nil`; inbound values never leak through.

Partition deliveries are started concurrently, one goroutine per partition, in the order in which each partition was first seen in the batch. Every partition is delivered even if another one fails; the errors of all failed deliveries are joined and returned. The inbound context is passed through without a derived cancellation, so downstream consumers that keep the context after returning are not affected.

When every item in a batch maps to the same partition, the batch is forwarded as-is without copying. When a batch is split, items are moved (not copied) into the partitions, so the processor declares that it mutates data; if the pipeline fans out to several consumers, the collector clones the batch before it reaches this processor.

## Using the metadata downstream

The processor only adds the partition keys to the request metadata; another component has to read them. A few things to keep in mind when wiring that up:

* **Exporter batching:** when an exporter batches with `sending_queue::batch`, requests from different partitions are merged, and only the metadata keys listed in `sending_queue::batch::partition::metadata_keys` are kept. List every partition key that a downstream component reads there, which also makes the exporter batch each partition separately. Without batching, the exporter's in-memory queue keeps the request metadata.
* **Inbound metadata:** receivers only need `include_metadata: true` when a key expression reads inbound request metadata through `otelcol.client.metadata`. The keys this processor adds do not need it.
* **High-cardinality keys:** every partition is delivered concurrently, so a key with many distinct values in one batch results in as many concurrent downstream calls. Put an exporter `sending_queue` between this processor and slow or synchronous backends.

## Examples

### Partition by a resource attribute

```yaml theme={null}
processors:
  partitioning:
    keys:
      tenant_id: 'resource.attributes["tenant.id"]'
```

Each outgoing call carries `client.Metadata["tenant_id"] = ["<tenant value>"]`.

### Partition by several keys

```yaml theme={null}
processors:
  partitioning:
    keys:
      tenant_id: 'resource.attributes["tenant.id"]'
      severity:  'log.severity_text'
```

Log records with the same `(tenant.id, severity_text)` tuple are grouped together, and the outgoing metadata carries both keys.

### Set a per-tenant HTTP header

Split mixed-tenant batches and send each tenant's data with its own `X-Scope-OrgID` header, for example to a multi-tenant backend. The exporter batches per tenant, and the [`headers_setter`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/extension/headerssetterextension/README.md) extension reads the tenant from the request metadata.

```yaml theme={null}
extensions:
  headers_setter:
    headers:
      - action: upsert
        key: X-Scope-OrgID
        from_context: tenant_id

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  partitioning:
    keys:
      tenant_id: 'resource.attributes["tenant.id"]'

exporters:
  otlp_grpc:
    endpoint: backend:4317
    auth:
      authenticator: headers_setter
    sending_queue:
      batch:
        flush_timeout: 1s
        partition:
          metadata_keys: [tenant_id]

service:
  extensions: [headers_setter]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [partitioning]
      exporters: [otlp_grpc]
```

### Route to Kafka topics and set the record key

Write each service's spans to its own Kafka topic, keyed by trace ID, with the [Kafka exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/kafkaexporter/README.md) reading the topic and key from the request metadata. When batching is enabled, the Kafka exporter requires both keys in `sending_queue::batch::partition::metadata_keys`.

```yaml theme={null}
processors:
  partitioning:
    keys:
      kafka_topic: 'resource.attributes["service.name"]'
      kafka_key: 'span.trace_id.string'

exporters:
  kafka:
    brokers: [kafka:9092]
    traces:
      topic_from_metadata_key: kafka_topic
      message_key_from_metadata_key: kafka_key
    sending_queue:
      batch:
        partition:
          metadata_keys: [kafka_topic, kafka_key]

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [partitioning]
      exporters: [kafka]
```

Partitioning by trace ID creates one partition per trace in each batch, which is why the exporter batches them again per key.

### Partition by inbound request metadata

Use a header sent by the client, such as a gRPC or HTTP header, as a partition key. `otelcol.client.metadata["<header>"]` returns a `[]string`; select a single element with `[0]` to satisfy the string constraint. The receiver must set `include_metadata: true` for inbound headers to be available.

```yaml theme={null}
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        include_metadata: true

processors:
  partitioning:
    keys:
      tenant_id: 'otelcol.client.metadata["x-tenant-id"][0]'
```

## Attributes

| Attribute Name | Description | Type | Values |
| - | - | - | - |
| `error.type` | Why key evaluation failed. | string | `evaluation_failed`, `invalid_value_type` |

## Configuration

### Example Configuration

```yaml theme={null}
partitioning:
  # Required. Map of partition key name to an OTTL value expression.
  keys:
    tenant_id: 'resource.attributes["tenant.id"]'
    severity: 'log.severity_text'
partitioning/no_keys:
  keys: {}
partitioning/empty_expression:
  keys:
    tenant_id: ""
partitioning/case_collision:
  keys:
    Tenant: 'resource.attributes["a"]'
    tenant: 'resource.attributes["b"]'
```

***

*Last generated: 2026-11-09*
