> ## Documentation Index
> Fetch the complete documentation index at: https://otel.fyi/llms.txt
> Use this file to discover all available pages before exploring further.

# Dynamicsampling

> OpenTelemetry processor for Dynamicsampling

# Dynamicsampling Processor

![Status](https://img.shields.io/badge/status-development-orange)

**Maintainers:** [@MikeGoldsmith](https://github.com/MikeGoldsmith), [@VinozzZ](https://github.com/VinozzZ), [@jmacd](https://github.com/jmacd)

**Source:** [opentelemetry-collector-contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/dynamicsamplingprocessor)

## Supported Telemetry

![Traces](https://img.shields.io/badge/traces-development-orange)

## Overview

The Dynamic Sampling Processor performs adaptive tail-based trace sampling using first-match rules-based routing to dynamic samplers. Each sampler produces a known sample rate which is encoded as `ot=th` in W3C TraceState for correct downstream metric weighting.

## How it works

1. Spans are accumulated in memory, grouped by trace ID.
2. A trace becomes ready for evaluation when **either** of two events fires (whichever comes first):
   * a root span arrives (any span with an empty `ParentSpanID`), or
   * `trace_timeout` elapses since the first span of the trace arrived. This timer is set on first-seen and is never extended by subsequent spans, ensuring a predictable upper bound on buffer occupancy.
3. After the trigger fires, the processor pauses for `decision_delay` to let in-flight straggler spans land. The same delay applies regardless of which event fired the trigger.
4. Rules are then evaluated in order against the accumulated trace. The first rule whose conditions all match selects the sampler; once a rule is selected, its sampler's keep/drop decision is final and no later rules are considered. A rule with no conditions is a catch-all.
5. The matched sampler produces a sample rate (1-in-N).
6. The keep/drop decision is made deterministically by comparing the sample rate's threshold against the randomness of the trace ID, using the [OTel consistent probability sampling](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/) algorithm.
7. Sampled traces are forwarded with two annotations on every span:
   * `otelcol.processor.dynamic_sampling.rule`: the name of the matched rule
   * W3C TraceState `ot=th:<hex>`: the threshold encoding the effective sample rate
8. The decision (sampled or dropped) is recorded in a per-trace LRU cache so that any late-arriving spans for the same trace ID are handled consistently with the original decision (see [Decision cache](#decision-cache) below).

## Configuration

```yaml theme={null}
processors:
  dynamic_sampling:
    # Upper bound on time a trace can sit in the buffer before its decision is
    # forced. Acts as a safety net for traces that never see a root span.
    trace_timeout: 30s

    # Pause between a triggering event (root span arrival or trace timeout) and
    # the actual decision evaluation. Allows in-flight straggler spans to land
    # before the trace is decided.
    decision_delay: 2s

    # Maximum number of traces held in the in-memory buffer.
    num_traces: 50000

    decision_cache:
      sampled_cache_size: 10000      # 0 disables the sampled cache
      non_sampled_cache_size: 10000  # 0 disables the not-sampled cache

    rules:
      # First-match: the first rule whose conditions all match selects the sampler.
      - name: keep-errors
        conditions:
          - "status.code == 2"
        sampler:
          type: always_sample

      - name: payment-service
        conditions:
          - 'resource.attributes["service.name"] == "payment"'
        sampler:
          type: ema_dynamic
          ema_dynamic:
            goal_sampling_percentage: 5
            key_fields: ["http.method", "http.route"]
            adjustment_interval: 15s
            weight: 0.5

      # Catch-all: no conditions, always matches.
      - name: default
        sampler:
          type: ema_dynamic
          ema_dynamic:
            goal_sampling_percentage: 20
            key_fields: ["service.name", "http.status_code"]
```

### Rules

Rules are evaluated in order; the first whose conditions all match selects the sampler. A rule with no conditions is a catch-all. Once a rule is selected, its sampler's decision is final: a drop stays dropped, the trace is not handed to any later rule.

> \[!WARNING]
> A rule with no conditions (a catch-all) placed before another rule consumes every trace and renders the later rules unreachable. The processor logs a warning at startup when it detects this configuration so it shows up in collector logs.

The intended pattern is specific-conditions rules first, catch-all last:

```yaml theme={null}
rules:
  - name: keep-errors
    conditions: ["status.code == 2"]
    sampler:
      type: always_sample
  - name: default              # catch-all
    sampler:
      type: ema_dynamic
      ema_dynamic:
        goal_sampling_percentage: 10
        key_fields: ["service.name"]
```

With the order above, error traces are always kept and every other trace is decided by `ema_dynamic`. Flipping the order so `default` comes first would mean `default` swallows every trace (including errors) and `keep-errors` is never reached, which is what the startup warning flags.

Each condition is a simple expression. In this initial release the supported forms are:

| Expression         | Meaning                                                      |
| ------------------ | ------------------------------------------------------------ |
| `field == value`   | Any span (or its resource) has `field` set to `value`.       |
| `field != value`   | No span has `field` set to `value`.                          |
| `status.code == N` | Any span has status code `N` (0 = Unset, 1 = Ok, 2 = Error). |
| `status.code != N` | No span has status code `N`.                                 |

OTTL-based conditions are planned for a follow-up release.

### Samplers

| Type                  | Behaviour                                                                               |
| --------------------- | --------------------------------------------------------------------------------------- |
| `always_sample`       | Keep every matching trace (sample rate 1).                                              |
| `deterministic`       | Fixed-rate probabilistic sampling using `sampling_percentage` (0, 100].                 |
| `ema_dynamic`         | EMA-based adaptive sampling targeting an average sampling percentage across keys.       |
| `ema_throughput`      | EMA-based adaptive sampling targeting a fixed events-per-second throughput across keys. |
| `windowed_throughput` | Sliding-window throughput sampler with separate update and lookback frequencies.        |

The adaptive samplers (`ema_dynamic`, `ema_throughput`, `windowed_throughput`) are backed by [`dynsampler-go`](https://github.com/honeycombio/dynsampler-go).

#### `deterministic`

```yaml theme={null}
sampler:
  type: deterministic
  deterministic:
    sampling_percentage: 10   # keep 10% of traces
```

#### `ema_dynamic`

Adapts the sample rate per traffic key over time, keeping a target average sampling percentage across all keys.

```yaml theme={null}
sampler:
  type: ema_dynamic
  ema_dynamic:
    goal_sampling_percentage: 10            # target % across all keys
    key_fields: ["service.name", "http.status_code"]
    adjustment_interval: 15s                # how often the EMA recalculates
    weight: 0.5                             # EMA weighting factor in [0, 1)
    max_keys: 500                           # 0 = unlimited
```

#### `ema_throughput`

Adjusts rates per key to hit a target sustained throughput in events per second.

```yaml theme={null}
sampler:
  type: ema_throughput
  ema_throughput:
    goal_throughput_per_sec: 100            # target events/sec across all keys
    key_fields: ["service.name", "http.status_code"]
    initial_sampling_rate: 10               # optional: rate used before first adjustment
    adjustment_interval: 15s
    weight: 0.5
    max_keys: 500
```

#### `windowed_throughput`

Sliding-window throughput sampler that decouples how often rates are recalculated (`update_frequency`) from the historical window used for the calculation (`lookback_frequency`). This reacts to traffic shifts faster than the EMA samplers at the cost of being more sensitive to short-term spikes.

```yaml theme={null}
sampler:
  type: windowed_throughput
  windowed_throughput:
    goal_throughput_per_sec: 100
    key_fields: ["service.name", "http.status_code"]
    update_frequency: 1s                    # how often rates recalculate
    lookback_frequency: 30s                 # historical window used in the calculation
    max_keys: 500
```

### Sampling keys

For samplers that accept `key_fields`, the sampling key for a trace is built by collecting distinct values of each `key_field` (across resource and span attributes), sorting them, and joining with the `•` separator. Missing fields are replaced with `<missing>`.

## Decision cache

When a trace's decision is finalised, the trace ID and outcome are recorded in
one of two LRU caches. Any later spans that arrive for that trace ID short-circuit
the accumulation path and are handled consistently with the original decision:

* **Sampled cache** (`decision_cache.sampled_cache_size`): late spans are forwarded
  immediately, stamped with the same rule attribute and `ot=th` TraceState as the
  original batch.
* **Not-sampled cache** (`decision_cache.non_sampled_cache_size`): late spans are
  silently dropped.

Setting a cache size to `0` disables that side of the cache. Late spans for a
decision class whose cache is disabled fall through to the normal pending-trace
path and may produce a second (possibly inconsistent) decision; with both caches
disabled, every late span falls through. Operators tune the cache sizes against
observed late-span volume.

### Buffer overflow vs decision cache

The in-memory accumulation buffer is sized by `num_traces`. If the buffer is full
when a span for a brand-new trace arrives, the processor evicts an existing
pending trace to make room; the evicted trace is dropped without a sampling
decision and without `ot=th` annotations, and is counted on
`processor_dynamic_sampling_traces_evicted`. The decision cache
(`decision_cache.*`) is a separate structure that records the outcome of
*completed* decisions for late-arriving spans, it does not protect pending traces
from eviction. Operators sizing the processor should watch `traces_evicted` and
increase `num_traces` if it is non-zero in steady state.

## Deployment considerations

The processor accumulates spans in memory, so all spans of a given trace must reach the same processor instance. Multi-instance deployments use the same two-tier pattern as the `tail_sampling` processor:

```
SDKs → Collectors (loadbalancing exporter, hash by traceID)
         → Collectors (dynamic_sampling processor)
           → Backend
```

## Relationship to `processor/tail_sampling`

This processor sits next to `tail_sampling` rather than extending it. The two are close in shape (both buffer traces and decide once a trigger fires) but use different evaluation models, and `tail_sampling`'s current model has several mechanical features that a rate-bearing sampler does not fit cleanly into.

### Decision semantics

In `tail_sampling`'s [policy loop](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/processor.go#L800-L817) only two outcomes short-circuit evaluation: any policy returning `Dropped`, and (when `sample_on_first_match` is enabled) the first policy to return `Sampled`. Every other outcome, including `NotSampled`, lets the loop continue to subsequent policies.

All common "this trace did not pass my check" votes from existing policies (probabilistic, status\_code, rate\_limiting, and\_policy, etc.) return `NotSampled`, never `Dropped`. `Dropped` is reserved for the explicit [`drop` policy](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/internal/sampling/drop.go#L46), and the [final-decision composition](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/processor.go#L821-L836) gives `Dropped` precedence over `Sampled`.

A rate-bearing sampler dropped into this model has to pick a return value for a probabilistically-dropped trace, and both options break correctness:

* Returning `NotSampled` matches the existing convention but does not stop the loop. A later policy voting `Sampled` would still cause the trace to be kept, and the rate-bearing sampler's view of what it controlled diverges from reality. Its rate calculations drift over time.
* Returning `Dropped` stops the loop, but it also wins precedence over every `Sampled` vote in the composition step. An operator pairing a rate-bearing sampler with an explicit `keep-errors` policy would expect errors to always win; instead the rate-bearing sampler's probabilistic drop would override the keep, inverting the configured intent.

### No rate or threshold in the policy contract

The [`Evaluator` interface](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/pkg/samplingpolicy/samplingpolicy.go#L78-L85) returns only a decision enum. There is no way for a policy to communicate the sample rate or threshold it applied, which means `ot=th` cannot be emitted from inside `tail_sampling` without expanding the contract. The processor has roughly twenty existing policies; widening the interface would touch all of them.

### `sample_on_first_match` would become correctness-load-bearing for one policy type only

Today `sample_on_first_match` is an opt-in optimization. Adding rate-bearing samplers as policies would make it mandatory for configurations using them; without it, the OR composition above produces drift. Existing `tail_sampling` users adding a rate-bearing policy without flipping the flag would silently get wrong rates. Validation cannot easily reject this because the flag is fine in isolation, and there is no current marker on a policy type that says "this policy requires first-match for correctness."

### Rule attribution

This processor records the matched rule name on every span in a sampled trace, which is only meaningful under first-match semantics. Under `tail_sampling`'s multi-policy OR model multiple policies can vote `Sampled` and there is no defined "winner" to attribute the trace to.

### Relationship to [PR #48865](https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/48865)

In-flight work on adding tracestate handling to `tail_sampling`'s probabilistic policy is input-driven: it reads SDK-supplied `ot=th` to adjust the tail probabilistic threshold (equalizing mode). This processor is output-driven: it computes a tail-stage rate and emits `ot=th`. The two address different problems and can ship independently.

For these reasons `dynamic_sampling` is a separate processor. The `tail_sampling` users retain the existing multi-policy model unchanged, and `dynamic_sampling` keeps a single evaluation model (first-match with rate-bearing samplers) end to end.

## Metrics

| Metric                                                    | Type      | Labels    | Description                                                                                            |
| --------------------------------------------------------- | --------- | --------- | ------------------------------------------------------------------------------------------------------ |
| `otelcol_processor_dynamic_sampling_traces_active`        | Gauge     |           | Traces currently in the accumulation buffer.                                                           |
| `otelcol_processor_dynamic_sampling_traces_sampled`       | Counter   | `rule`    | Traces kept, attributed to the rule that selected them.                                                |
| `otelcol_processor_dynamic_sampling_traces_dropped`       | Counter   | `rule`    | Traces dropped, attributed to the rule that selected them.                                             |
| `otelcol_processor_dynamic_sampling_decision_sample_rate` | Histogram | `rule`    | Distribution of effective sample rates produced per rule.                                              |
| `otelcol_processor_dynamic_sampling_decision_triggers`    | Counter   | `trigger` | Number of trace decisions made, labelled by which event triggered them (`root_span`, `trace_timeout`). |
| `otelcol_processor_dynamic_sampling_traces_evicted`       | Counter   |           | Traces evicted from the buffer before a decision could be made.                                        |

## Output attributes

Every span in a sampled trace is annotated with:

| Attribute                                 | Type   | Example       | Description                                |
| ----------------------------------------- | ------ | ------------- | ------------------------------------------ |
| `otelcol.processor.dynamic_sampling.rule` | string | `keep-errors` | Name of the rule that selected this trace. |

The sample rate is encoded in W3C TraceState as `ot=th:<hex>` per the OTel consistent probability sampling spec. The `spanmetrics` connector (`enable_metrics_sampling_method: true`) reads this field to produce correctly weighted R.E.D metrics from sampled data.

## Future work

* OTTL-based conditions
* Shared-storage backed scaling for single-tier deployments
* Rule and sampler hot-reload

***

*Last generated: 2026-07-06*