Dynamicsampling Processor

Maintainers: @MikeGoldsmith, @VinozzZ, @jmacd Source: opentelemetry-collector-contrib

Supported Telemetry

Overview

The Dynamic Sampling Processor performs adaptive tail-based trace sampling using first-match rules-based routing to dynamic samplers. Each sampler produces a known sample rate which is encoded as ot=th in W3C TraceState for correct downstream metric weighting.

How it works

Spans are accumulated in memory, grouped by trace ID.
A trace becomes ready for evaluation when either of two events fires (whichever comes first):
- a root span arrives (any span with an empty ParentSpanID), or
- trace_timeout elapses since the first span of the trace arrived. This timer is set on first-seen and is never extended by subsequent spans, ensuring a predictable upper bound on buffer occupancy.
After the trigger fires, the processor pauses for decision_delay to let in-flight straggler spans land. The same delay applies regardless of which event fired the trigger.
Rules are then evaluated in order against the accumulated trace. The first rule whose conditions all match selects the sampler; once a rule is selected, its sampler’s keep/drop decision is final and no later rules are considered. A rule with no conditions is a catch-all.
The matched sampler produces a sample rate (1-in-N).
The keep/drop decision is made deterministically by comparing the sample rate’s threshold against the randomness of the trace ID, using the OTel consistent probability sampling algorithm.
Sampled traces are forwarded with two annotations on every span:
- otelcol.processor.dynamic_sampling.rule: the name of the matched rule
- W3C TraceState ot=th:<hex>: the threshold encoding the effective sample rate
The decision (sampled or dropped) is recorded in a per-trace LRU cache so that any late-arriving spans for the same trace ID are handled consistently with the original decision (see Decision cache below).

Configuration

processors:
  dynamic_sampling:
    # Upper bound on time a trace can sit in the buffer before its decision is
    # forced. Acts as a safety net for traces that never see a root span.
    trace_timeout: 30s

    # Pause between a triggering event (root span arrival or trace timeout) and
    # the actual decision evaluation. Allows in-flight straggler spans to land
    # before the trace is decided.
    decision_delay: 2s

    # Maximum number of traces held in the in-memory buffer.
    num_traces: 50000

    decision_cache:
      sampled_cache_size: 10000      # 0 disables the sampled cache
      non_sampled_cache_size: 10000  # 0 disables the not-sampled cache

    rules:
      # First-match: the first rule whose conditions all match selects the sampler.
      - name: keep-errors
        conditions:
          - "status.code == 2"
        sampler:
          type: always_sample

      - name: payment-service
        conditions:
          - 'resource.attributes["service.name"] == "payment"'
        sampler:
          type: ema_dynamic
          ema_dynamic:
            goal_sampling_percentage: 5
            key_fields: ["http.method", "http.route"]
            adjustment_interval: 15s
            weight: 0.5

      # Catch-all: no conditions, always matches.
      - name: default
        sampler:
          type: ema_dynamic
          ema_dynamic:
            goal_sampling_percentage: 20
            key_fields: ["service.name", "http.status_code"]

Rules

Rules are evaluated in order; the first whose conditions all match selects the sampler. A rule with no conditions is a catch-all. Once a rule is selected, its sampler’s decision is final: a drop stays dropped, the trace is not handed to any later rule.

[!WARNING] A rule with no conditions (a catch-all) placed before another rule consumes every trace and renders the later rules unreachable. The processor logs a warning at startup when it detects this configuration so it shows up in collector logs.

The intended pattern is specific-conditions rules first, catch-all last:

rules:
  - name: keep-errors
    conditions: ["status.code == 2"]
    sampler:
      type: always_sample
  - name: default              # catch-all
    sampler:
      type: ema_dynamic
      ema_dynamic:
        goal_sampling_percentage: 10
        key_fields: ["service.name"]

With the order above, error traces are always kept and every other trace is decided by ema_dynamic. Flipping the order so default comes first would mean default swallows every trace (including errors) and keep-errors is never reached, which is what the startup warning flags. Each condition is a simple expression. In this initial release the supported forms are:

Expression	Meaning
`field == value`	Any span (or its resource) has `field` set to `value`.
`field != value`	No span has `field` set to `value`.
`status.code == N`	Any span has status code `N` (0 = Unset, 1 = Ok, 2 = Error).
`status.code != N`	No span has status code `N`.

OTTL-based conditions are planned for a follow-up release.

Samplers

Type	Behaviour
`always_sample`	Keep every matching trace (sample rate 1).
`deterministic`	Fixed-rate probabilistic sampling using `sampling_percentage` (0, 100].
`ema_dynamic`	EMA-based adaptive sampling targeting an average sampling percentage across keys.
`ema_throughput`	EMA-based adaptive sampling targeting a fixed events-per-second throughput across keys.
`windowed_throughput`	Sliding-window throughput sampler with separate update and lookback frequencies.

The adaptive samplers (ema_dynamic, ema_throughput, windowed_throughput) are backed by dynsampler-go.

`deterministic`

sampler:
  type: deterministic
  deterministic:
    sampling_percentage: 10   # keep 10% of traces

`ema_dynamic`

Adapts the sample rate per traffic key over time, keeping a target average sampling percentage across all keys.

sampler:
  type: ema_dynamic
  ema_dynamic:
    goal_sampling_percentage: 10            # target % across all keys
    key_fields: ["service.name", "http.status_code"]
    adjustment_interval: 15s                # how often the EMA recalculates
    weight: 0.5                             # EMA weighting factor in [0, 1)
    max_keys: 500                           # 0 = unlimited

`ema_throughput`

Adjusts rates per key to hit a target sustained throughput in events per second.

sampler:
  type: ema_throughput
  ema_throughput:
    goal_throughput_per_sec: 100            # target events/sec across all keys
    key_fields: ["service.name", "http.status_code"]
    initial_sampling_rate: 10               # optional: rate used before first adjustment
    adjustment_interval: 15s
    weight: 0.5
    max_keys: 500

`windowed_throughput`

Sliding-window throughput sampler that decouples how often rates are recalculated (update_frequency) from the historical window used for the calculation (lookback_frequency). This reacts to traffic shifts faster than the EMA samplers at the cost of being more sensitive to short-term spikes.

sampler:
  type: windowed_throughput
  windowed_throughput:
    goal_throughput_per_sec: 100
    key_fields: ["service.name", "http.status_code"]
    update_frequency: 1s                    # how often rates recalculate
    lookback_frequency: 30s                 # historical window used in the calculation
    max_keys: 500

Sampling keys

For samplers that accept key_fields, the sampling key for a trace is built by collecting distinct values of each key_field (across resource and span attributes), sorting them, and joining with the • separator. Missing fields are replaced with <missing>.

Decision cache

When a trace’s decision is finalised, the trace ID and outcome are recorded in one of two LRU caches. Any later spans that arrive for that trace ID short-circuit the accumulation path and are handled consistently with the original decision:

Sampled cache (decision_cache.sampled_cache_size): late spans are forwarded immediately, stamped with the same rule attribute and ot=th TraceState as the original batch.
Not-sampled cache (decision_cache.non_sampled_cache_size): late spans are silently dropped.

Setting a cache size to 0 disables that side of the cache. Late spans for a decision class whose cache is disabled fall through to the normal pending-trace path and may produce a second (possibly inconsistent) decision; with both caches disabled, every late span falls through. Operators tune the cache sizes against observed late-span volume.

Buffer overflow vs decision cache

The in-memory accumulation buffer is sized by num_traces. If the buffer is full when a span for a brand-new trace arrives, the processor evicts an existing pending trace to make room; the evicted trace is dropped without a sampling decision and without ot=th annotations, and is counted on processor_dynamic_sampling_traces_evicted. The decision cache (decision_cache.*) is a separate structure that records the outcome of completed decisions for late-arriving spans, it does not protect pending traces from eviction. Operators sizing the processor should watch traces_evicted and increase num_traces if it is non-zero in steady state.

Deployment considerations

The processor accumulates spans in memory, so all spans of a given trace must reach the same processor instance. Multi-instance deployments use the same two-tier pattern as the tail_sampling processor:

SDKs → Collectors (loadbalancing exporter, hash by traceID)
         → Collectors (dynamic_sampling processor)
           → Backend

Relationship to `processor/tail_sampling`

This processor sits next to tail_sampling rather than extending it. The two are close in shape (both buffer traces and decide once a trigger fires) but use different evaluation models, and tail_sampling’s current model has several mechanical features that a rate-bearing sampler does not fit cleanly into.

Decision semantics

In tail_sampling’s policy loop only two outcomes short-circuit evaluation: any policy returning Dropped, and (when sample_on_first_match is enabled) the first policy to return Sampled. Every other outcome, including NotSampled, lets the loop continue to subsequent policies. All common “this trace did not pass my check” votes from existing policies (probabilistic, status_code, rate_limiting, and_policy, etc.) return NotSampled, never Dropped. Dropped is reserved for the explicit drop policy, and the final-decision composition gives Dropped precedence over Sampled. A rate-bearing sampler dropped into this model has to pick a return value for a probabilistically-dropped trace, and both options break correctness:

Returning NotSampled matches the existing convention but does not stop the loop. A later policy voting Sampled would still cause the trace to be kept, and the rate-bearing sampler’s view of what it controlled diverges from reality. Its rate calculations drift over time.
Returning Dropped stops the loop, but it also wins precedence over every Sampled vote in the composition step. An operator pairing a rate-bearing sampler with an explicit keep-errors policy would expect errors to always win; instead the rate-bearing sampler’s probabilistic drop would override the keep, inverting the configured intent.

No rate or threshold in the policy contract

The Evaluator interface returns only a decision enum. There is no way for a policy to communicate the sample rate or threshold it applied, which means ot=th cannot be emitted from inside tail_sampling without expanding the contract. The processor has roughly twenty existing policies; widening the interface would touch all of them.

`sample_on_first_match` would become correctness-load-bearing for one policy type only

Today sample_on_first_match is an opt-in optimization. Adding rate-bearing samplers as policies would make it mandatory for configurations using them; without it, the OR composition above produces drift. Existing tail_sampling users adding a rate-bearing policy without flipping the flag would silently get wrong rates. Validation cannot easily reject this because the flag is fine in isolation, and there is no current marker on a policy type that says “this policy requires first-match for correctness.”

Rule attribution

This processor records the matched rule name on every span in a sampled trace, which is only meaningful under first-match semantics. Under tail_sampling’s multi-policy OR model multiple policies can vote Sampled and there is no defined “winner” to attribute the trace to.

Relationship to PR #48865

In-flight work on adding tracestate handling to tail_sampling’s probabilistic policy is input-driven: it reads SDK-supplied ot=th to adjust the tail probabilistic threshold (equalizing mode). This processor is output-driven: it computes a tail-stage rate and emits ot=th. The two address different problems and can ship independently. For these reasons dynamic_sampling is a separate processor. The tail_sampling users retain the existing multi-policy model unchanged, and dynamic_sampling keeps a single evaluation model (first-match with rate-bearing samplers) end to end.

Metrics

Metric	Type	Labels	Description
`otelcol_processor_dynamic_sampling_traces_active`	Gauge		Traces currently in the accumulation buffer.
`otelcol_processor_dynamic_sampling_traces_sampled`	Counter	`rule`	Traces kept, attributed to the rule that selected them.
`otelcol_processor_dynamic_sampling_traces_dropped`	Counter	`rule`	Traces dropped, attributed to the rule that selected them.
`otelcol_processor_dynamic_sampling_decision_sample_rate`	Histogram	`rule`	Distribution of effective sample rates produced per rule.
`otelcol_processor_dynamic_sampling_decision_triggers`	Counter	`trigger`	Number of trace decisions made, labelled by which event triggered them (`root_span`, `trace_timeout`).
`otelcol_processor_dynamic_sampling_traces_evicted`	Counter		Traces evicted from the buffer before a decision could be made.

Output attributes

Every span in a sampled trace is annotated with:

Attribute	Type	Example	Description
`otelcol.processor.dynamic_sampling.rule`	string	`keep-errors`	Name of the rule that selected this trace.

The sample rate is encoded in W3C TraceState as ot=th:<hex> per the OTel consistent probability sampling spec. The spanmetrics connector (enable_metrics_sampling_method: true) reads this field to produce correctly weighted R.E.D metrics from sampled data.

Future work

OTTL-based conditions
Shared-storage backed scaling for single-tier deployments
Rule and sampler hot-reload

Last generated: 2026-07-06

​Dynamicsampling Processor

​Supported Telemetry

​Overview

​How it works

​Configuration

​Rules

​Samplers

​deterministic

​ema_dynamic

​ema_throughput

​windowed_throughput

​Sampling keys

​Decision cache

​Buffer overflow vs decision cache

​Deployment considerations

​Relationship to processor/tail_sampling

​Decision semantics

​No rate or threshold in the policy contract

​sample_on_first_match would become correctness-load-bearing for one policy type only

​Rule attribution

​Relationship to PR #48865

​Metrics

​Output attributes

​Future work