Dynamicsampling Processor

Maintainers: @MikeGoldsmith, @VinozzZ, @jmacd Source: opentelemetry-collector-contrib

Supported Telemetry

Overview

The Dynamic Sampling Processor performs adaptive tail-based trace sampling using first-match rules-based routing to dynamic samplers. Each sampler produces a known sample rate which is encoded as ot=th in W3C TraceState for correct downstream metric weighting.

How it works

Spans are accumulated in memory, grouped by trace ID.
A trace becomes ready for evaluation when either of two events fires (whichever comes first):
- a root span arrives (any span with an empty ParentSpanID), or
- trace_timeout elapses since the first span of the trace arrived. This timer is set on first-seen and is never extended by subsequent spans, ensuring a predictable upper bound on buffer occupancy.
After the trigger fires, the processor pauses for decision_delay to let in-flight straggler spans land. The same delay applies regardless of which event fired the trigger.
Rules are then evaluated in order against the accumulated trace. The first rule whose conditions all match selects the sampler; once a rule is selected, its sampler’s keep/drop decision is final and no later rules are considered. A rule with no conditions is a catch-all.
The matched sampler produces a sample rate (1-in-N). Samplers only ever produce rates; no sampler makes a keep/drop decision itself.
The processor converts the rate to a threshold (a rate of N becomes the threshold encoding probability 1/N) and makes the keep/drop decision by comparing that threshold against the trace’s randomness (ot=rv when present, otherwise derived from the trace ID), using the OTel consistent probability sampling algorithm.

[!NOTE] The adaptive samplers come from dynsampler-go, which the processor uses only to compute rates. dynsampler-go’s own samplers can make keep/drop decisions internally (its DeterministicSampler applies its own hash-based check, for example), but this processor never uses that path: the rate-to-threshold conversion and the randomness comparison in step 6 are the single decision mechanism for every sampler type, including this processor’s deterministic type. This is what makes decisions reproducible for a given trace and correctly weighted downstream via ot=th.

Sampled traces are forwarded with two annotations on every span:
- otelcol.processor.dynamic_sampling.rule: the name of the matched rule
- W3C TraceState ot=th:<hex>: the threshold encoding the effective sample rate
The decision (sampled or dropped) is recorded in a per-trace LRU cache so that any late-arriving spans for the same trace ID are handled consistently with the original decision (see Decision cache below).

Configuration

processors:
  dynamic_sampling:
    # Upper bound on time a trace can sit in the buffer before its decision is
    # forced. Acts as a safety net for traces that never see a root span.
    trace_timeout: 30s

    # Pause between a triggering event (root span arrival or trace timeout) and
    # the actual decision evaluation. Allows in-flight straggler spans to land
    # before the trace is decided.
    decision_delay: 2s

    # Maximum number of traces held in the in-memory buffer.
    num_traces: 50000

    decision_cache:
      sampled_cache_size: 10000      # 0 disables the sampled cache
      non_sampled_cache_size: 10000  # 0 disables the not-sampled cache

    rules:
      # First-match: the first rule whose conditions all match selects the sampler.
      - name: keep-errors
        conditions:
          - "status.code == 2"
        sampler:
          type: always_sample

      - name: payment-service
        conditions:
          - 'resource.attributes["service.name"] == "payment"'
        sampler:
          type: ema_dynamic
          goal_sampling_percentage: 5
          key_attributes: ["http.method", "http.route"]
          adjustment_interval: 15s
          weight: 0.5

      # Catch-all: no conditions, always matches.
      - name: default
        sampler:
          type: ema_dynamic
          goal_sampling_percentage: 20
          key_attributes: ["service.name", "http.status_code"]

    # Optional. OTTL boolean expression evaluated per span. When it returns true
    # for any span in the trace, the trace transitions from accumulation to the
    # decision-delay phase without waiting for trace_timeout. Defaults to
    # IsRootSpan() when unset.
    # root_span_condition: 'IsRootSpan() or span.attributes["otelcol.dynamic_sampling.root_span"] == true'

Root-span detection

The processor moves a trace out of accumulation and into the decision-delay phase as soon as it sees a span it considers a “root”. By default that means IsRootSpan(): any span whose W3C ParentSpanID is empty. This works for producer-side head-sampled traces where the true root always lands at the same collector, but it stalls when:

Only the server side of a cross-process trace reaches this collector, so no observed span has an empty parent.
The operator wants to trigger on a producer-supplied hint (e.g. a message-broker consumer or batch-job entry point) rather than on the transport-level root.

root_span_condition lets the operator override the trigger with any OTTL boolean expression evaluated in the ottlspan context. The first span for which it returns true starts the decision-delay timer. Producer hint (fires on any real root, or on any span the producer explicitly tagged):

root_span_condition: 'IsRootSpan() or span.attributes["otelcol.dynamic_sampling.root_span"] == true'

Cross-process server (accept the gateway’s server span as a trigger even though it has a parent):

root_span_condition: 'IsRootSpan() or (span.kind == SPAN_KIND_SERVER and resource.attributes["service.name"] == "gateway")'

Message consumer (trigger on the consumer side of a queue rather than waiting for trace_timeout):

root_span_condition: 'IsRootSpan() or (span.kind == SPAN_KIND_CONSUMER and IsMatch(span.name, "^receive "))'

Evaluation errors are counted on processor_dynamic_sampling_ottl_eval_errors under the sentinel rule="_root_span_condition" label so they show up separately from rule-condition errors. A span whose evaluation errored is treated as non-matching, so a broken expression can only delay the trigger to trace_timeout, never fire it prematurely.

Rules

Rules are evaluated in order; the first whose conditions all match selects the sampler. A rule with no conditions is a catch-all. Once a rule is selected, its sampler’s decision is final: a drop stays dropped, the trace is not handed to any later rule. Each rule’s name must be unique. It is recorded on the decision metrics (the rule attribute) and stamped on sampled spans, so pick names you want to see on dashboards. Names starting with _ are rejected at config validation: that prefix is reserved for processor-internal decision labels (such as _eviction), so user rules can never collide with them.

[!WARNING] A rule with no conditions (a catch-all) placed before another rule consumes every trace and renders the later rules unreachable. The processor logs a warning at startup when it detects this configuration so it shows up in collector logs.

Conditions are OTTL boolean expressions evaluated in the ottlspan context, which gives access to the span, its resource, and its instrumentation scope. Path expressions must be qualified with the context they refer to: span.attributes["k"], resource.attributes["k"], span.status.code, and so on. See OTTL Boolean Expressions for the full grammar (including and, or, and parentheses), and the ottlspan paths reference for the list of accessible fields. The intended pattern is specific-conditions rules first, catch-all last:

rules:
  - name: keep-errors
    conditions:
      - span.status.code == STATUS_CODE_ERROR
    sampler:
      type: always_sample
  - name: default              # catch-all
    sampler:
      type: ema_dynamic
      goal_sampling_percentage: 10
      key_attributes: ["service.name"]

With the order above, error traces are always kept and every other trace is decided by ema_dynamic. Flipping the order so default comes first would mean default swallows every trace (including errors) and keep-errors is never reached, which is what the startup warning flags.

How multiple conditions combine

Each rule can carry multiple OTTL conditions. The match field controls how they are combined against the spans of the accumulated trace:

match: any_span (default): each condition must be satisfied by some span in the trace, but not necessarily the same span across conditions. Reads as “the trace has these characteristics.”
match: same_span: some single span in the trace must satisfy all conditions at once. Reads as “there is a span with these characteristics.”

match is not permitted on catch-all rules (rules with no conditions). Example: keep a trace only when a single span is both an error and from the payment service:

- name: payment-errors
  match: same_span
  conditions:
    - span.status.code == STATUS_CODE_ERROR
    - resource.attributes["service.name"] == "payment"
  sampler:
    type: always_sample

Example (default any_span): keep any trace that touched the payment service and saw an error somewhere:

- name: payment-involved-errors
  conditions:
    - span.status.code == STATUS_CODE_ERROR
    - resource.attributes["service.name"] == "payment"
  sampler:
    type: always_sample

If an OTTL condition raises an evaluation error at runtime (for example, a path does not exist on a specific span) the condition is treated as false for that span and the otelcol_processor_dynamic_sampling_ottl_eval_errors counter is incremented, labelled by the rule name. Other spans and conditions continue to evaluate normally.

Condition examples

Keep every trace where an HTTP server span returned 5xx, using OTTL’s inline and:

- name: http-5xx
  match: same_span
  conditions:
    - span.kind == SPAN_KIND_SERVER and span.attributes["http.response.status_code"] >= 500
  sampler:
    type: always_sample

Keep every trace that includes a root span from a specific service, using OTTL’s IsRootSpan() helper:

- name: root-checkout
  match: same_span
  conditions:
    - IsRootSpan() and resource.attributes["service.name"] == "checkout"
  sampler:
    type: always_sample

Keep traces from either the checkout or the billing service, at a modest adaptive rate:

- name: high-value-services
  conditions:
    - resource.attributes["service.name"] == "checkout" or resource.attributes["service.name"] == "billing"
  sampler:
    type: ema_dynamic
    goal_sampling_percentage: 25
    key_attributes: ["service.name", "http.route"]

Match on span name prefix using OTTL’s IsMatch regex helper:

- name: api-calls
  match: same_span
  conditions:
    - IsMatch(span.name, "^GET /api/")
  sampler:
    type: deterministic
    deterministic:
      sampling_percentage: 5

For the full set of comparison operators, string functions, and converter helpers available in conditions, see OTTL Functions.

Samplers

Type	Behaviour
`always_sample`	Keep every matching trace (sample rate 1).
`deterministic`	Fixed-rate probabilistic sampling using `sampling_percentage` (0, 100].
`ema_dynamic`	EMA-based adaptive sampling targeting an average sampling percentage across keys.
`ema_throughput`	EMA-based adaptive sampling targeting a fixed events-per-second throughput across keys.
`windowed_throughput`	Sliding-window throughput sampler with separate update and lookback frequencies.

The adaptive samplers (ema_dynamic, ema_throughput, windowed_throughput) are backed by dynsampler-go.

`deterministic`

sampler:
  type: deterministic
  sampling_percentage: 10   # keep 10% of traces

`ema_dynamic`

Adapts the sample rate per traffic key over time, keeping a target average sampling percentage across all keys.

sampler:
  type: ema_dynamic
  goal_sampling_percentage: 10            # target % across all keys
  key_attributes: ["service.name", "http.status_code"]
  adjustment_interval: 15s                # how often the EMA recalculates
  weight: 0.5                             # EMA weighting factor in [0, 1)
  max_keys: 500                           # 0 = unlimited

`ema_throughput`

Adjusts rates per key to hit a target sustained throughput in events per second.

sampler:
  type: ema_throughput
  goal_throughput_per_sec: 100            # target events/sec across all keys
  key_attributes: ["service.name", "http.status_code"]
  adjustment_interval: 15s
  weight: 0.5
  max_keys: 500

`windowed_throughput`

Sliding-window throughput sampler that decouples how often rates are recalculated (update_frequency) from the historical window used for the calculation (lookback_frequency). This reacts to traffic shifts faster than the EMA samplers at the cost of being more sensitive to short-term spikes.

sampler:
  type: windowed_throughput
  goal_throughput_per_sec: 100
  key_attributes: ["service.name", "http.status_code"]
  update_frequency: 1s                    # how often rates recalculate
  lookback_frequency: 30s                 # historical window used in the calculation
  max_keys: 500

Sampling keys

For samplers that accept key_attributes, the sampling key for a trace is built by collecting distinct values of each named attribute (across resource and span attributes), sorting them, and joining with the • separator. Missing attributes are replaced with <missing>.

Worked examples

Two complete configurations for the most common deployment shapes.

Error retention with an adaptive default

Keep every error trace, and let an adaptive sampler settle the rest at a target percentage per traffic class. This is the right starting shape for most single-instance deployments: errors are never lost, and the default rule adapts as traffic mix shifts.

processors:
  dynamic_sampling:
    trace_timeout: 30s
    decision_delay: 2s
    num_traces: 50000
    decision_cache:
      sampled_cache_size: 50000
      non_sampled_cache_size: 100000
    rules:
      # Errors are always kept, evaluated first.
      - name: keep-errors
        conditions:
          - span.status.code == STATUS_CODE_ERROR
        sampler:
          type: always_sample
      # Everything else settles at ~10% per (service, route) class, so
      # low-traffic routes stay visible while hot routes are downsampled.
      - name: default
        sampler:
          type: ema_dynamic
          goal_sampling_percentage: 10
          key_attributes: ["service.name", "http.route"]
          adjustment_interval: 15s
          weight: 0.5

Sizing guidance: num_traces bounds memory and should cover the number of traces that start within one trace_timeout window at peak. The decision caches should be a small multiple of that, since they only store trace IDs and outcomes; undersizing them turns late spans of decided traces back into new partial traces.

Throughput-bounded fleet

Cap the spans-per-second this collector emits regardless of incoming volume, useful when the downstream backend is provisioned for a fixed ingest rate. The throughput sampler continuously adjusts per-key rates to hold the total near the goal.

processors:
  dynamic_sampling:
    trace_timeout: 30s
    decision_delay: 2s
    num_traces: 100000
    decision_cache:
      sampled_cache_size: 100000
      non_sampled_cache_size: 200000
    # Under sustained overload, shed evicted traces in constant time instead
    # of paying rule evaluation for every one (see Buffer overflow and
    # eviction policy below).
    eviction:
      policy: probabilistic
      sampling_percentage: 10
    rules:
      - name: throughput-cap
        sampler:
          type: windowed_throughput
          goal_throughput_per_sec: 1000
          key_attributes: ["service.name"]
          update_frequency: 1s
          lookback_frequency: 30s

The goal is enforced per collector instance: a fleet of N instances emits up to N times the configured throughput, so divide the backend budget by the instance count. Keying by service.name means each service’s share adapts to its share of total traffic rather than being fixed.

Decision cache

When a trace’s decision is finalised, the trace ID and outcome are recorded in one of two LRU caches. Any later spans that arrive for that trace ID short-circuit the accumulation path and are handled consistently with the original decision:

Sampled cache (decision_cache.sampled_cache_size): late spans are forwarded immediately, stamped with the same rule attribute and ot=th TraceState as the original batch.
Not-sampled cache (decision_cache.non_sampled_cache_size): late spans are silently dropped.

Setting a cache size to 0 disables that side of the cache. Late spans for a decision class whose cache is disabled fall through to the normal pending-trace path and may produce a second (possibly inconsistent) decision; with both caches disabled, every late span falls through. Operators tune the cache sizes against observed late-span volume.

Buffer overflow and eviction policy

The in-memory accumulation buffer is sized by num_traces. When the buffer is full and spans for brand-new traces arrive, the processor evicts the oldest pending traces to make room (the buffer may transiently exceed num_traces by the number of new traces in a single incoming batch). Every evicted trace receives a real sampling decision immediately, with the spans seen so far and no decision_delay; the decision is recorded in the decision cache so late-arriving spans are handled consistently, and kept traces carry correct ot=th annotations. How that decision is made is configurable:

eviction:
  policy: evaluate            # evaluate (default) | probabilistic
  sampling_percentage: 10     # required for probabilistic

evaluate (default): the evicted trace runs through the normal rules and sampler path. Your rules keep working under pressure (e.g. a keep-errors rule still keeps error traces), at the cost of OTTL evaluation over every span of every evicted trace.
probabilistic: rule evaluation is skipped and the trace is decided by comparing a threshold derived from sampling_percentage against the trace’s randomness. Constant-time work per evicted trace regardless of trace size or rule count, so an overloaded instance sheds load instead of amplifying it. Kept traces are stamped with the corresponding ot=th (and the sentinel rule attribution _eviction), so downstream weighting stays accurate. Recommended for high-throughput deployments where eviction indicates genuine overload.

Note the decision may be made on an incomplete trace in both modes: spans still in flight when the trace is evicted are treated as late spans and follow the recorded decision. The decision cache (decision_cache.*) is a separate structure recording completed decisions; it does not protect pending traces from eviction. Operators sizing the processor should watch traces_evicted (every eviction) and decision_triggers{trigger="eviction"} and increase num_traces if they are non-zero in steady state.

Deployment considerations

The processor accumulates spans in memory, so all spans of a given trace must reach the same processor instance. Multi-instance deployments use the same two-tier pattern as the tail_sampling processor:

SDKs → Collectors (loadbalancing exporter, hash by traceID)
         → Collectors (dynamic_sampling processor)
           → Backend

Known limitations

Decisions are per-instance. All spans of a trace must reach the same instance; scale out with the two-tier loadbalancing pattern above. Shared storage for a single-tier deployment is future work.
The buffer is bounded by trace count, not bytes. Steady-state memory is approximately span rate x buffer window (decision_delay after the decision triggers, up to trace_timeout) x average span size, plus the decision caches. Trace shape barely matters: per-trace bookkeeping is small, so many small traces cost slightly more than a few giant ones at the same span rate. When num_traces is exceeded the oldest trace is evicted with a real decision (see the eviction policy section).
Shutdown drains rather than discards. Pending traces are decided with the spans seen so far and kept traces are forwarded, so a clean restart or rollout does not silently lose the buffer. A crash still loses it.
Rules are fixed at startup. Changing rules means a restart (which drains, see above). Hot reload is future work.
Traces only. Rule conditions use the OTTL span context, and the configuration is trace-scoped; correlated log sampling is future work and will add log-specific configuration alongside the existing fields.

Relationship to `processor/tail_sampling`

This processor sits next to tail_sampling rather than extending it. The two are close in shape (both buffer traces and decide once a trigger fires) but use different evaluation models, and tail_sampling’s current model has several mechanical features that a rate-bearing sampler does not fit cleanly into.

Decision semantics

In tail_sampling’s policy loop only two outcomes short-circuit evaluation: any policy returning Dropped, and (when sample_on_first_match is enabled) the first policy to return Sampled. Every other outcome, including NotSampled, lets the loop continue to subsequent policies. All common “this trace did not pass my check” votes from existing policies (probabilistic, status_code, rate_limiting, and_policy, etc.) return NotSampled, never Dropped. Dropped is reserved for the explicit drop policy, and the final-decision composition gives Dropped precedence over Sampled. A rate-bearing sampler dropped into this model has to pick a return value for a probabilistically-dropped trace, and both options break correctness:

Returning NotSampled matches the existing convention but does not stop the loop. A later policy voting Sampled would still cause the trace to be kept, and the rate-bearing sampler’s view of what it controlled diverges from reality. Its rate calculations drift over time.
Returning Dropped stops the loop, but it also wins precedence over every Sampled vote in the composition step. An operator pairing a rate-bearing sampler with an explicit keep-errors policy would expect errors to always win; instead the rate-bearing sampler’s probabilistic drop would override the keep, inverting the configured intent.

No rate or threshold in the policy contract

The Evaluator interface returns only a decision enum. There is no way for a policy to communicate the sample rate or threshold it applied, which means ot=th cannot be emitted from inside tail_sampling without expanding the contract. The processor has roughly twenty existing policies; widening the interface would touch all of them.

`sample_on_first_match` would become correctness-load-bearing for one policy type only

Today sample_on_first_match is an opt-in optimization. Adding rate-bearing samplers as policies would make it mandatory for configurations using them; without it, the OR composition above produces drift. Existing tail_sampling users adding a rate-bearing policy without flipping the flag would silently get wrong rates. Validation cannot easily reject this because the flag is fine in isolation, and there is no current marker on a policy type that says “this policy requires first-match for correctness.”

Rule attribution

This processor records the matched rule name on every span in a sampled trace, which is only meaningful under first-match semantics. Under tail_sampling’s multi-policy OR model multiple policies can vote Sampled and there is no defined “winner” to attribute the trace to.

Relationship to PR #48865

In-flight work on adding tracestate handling to tail_sampling’s probabilistic policy is input-driven: it reads SDK-supplied ot=th to adjust the tail probabilistic threshold (equalizing mode). This processor is output-driven: it computes a tail-stage rate and emits ot=th. The two address different problems and can ship independently. For these reasons dynamic_sampling is a separate processor. The tail_sampling users retain the existing multi-policy model unchanged, and dynamic_sampling keeps a single evaluation model (first-match with rate-bearing samplers) end to end.

Interoperability with upstream sampling

The processor honours any incoming ot=th (sampling threshold) and ot=rv (explicit randomness) already set on incoming spans by an upstream sampler, such as an SDK head sampler or a probabilistic collector processor:

Randomness. If an incoming span carries ot=rv, that value is used to make the sampling decision. Otherwise the last 7 bytes of the trace ID are used, per the consistent probability spec.
Population-relative rate (equalizing). The rule’s rate N is interpreted as the operator’s target for the original population: “keep 1-in-N of all traces before any sampling.” The effective absolute keep probability is min(P_upstream, 1/N). This is the same composition mode as equalizing in processor/probabilisticsamplerprocessor.
Threshold monotonicity. If a span already carries an ot=th stricter than what this processor would emit, the incoming value is preserved. This matches the consistent probability spec: a downstream stage may raise a threshold but never lower it.

Worked example: composing with a head sampler

processors:
  probabilistic_sampler:
    sampling_percentage: 50
    mode: equalizing # writes ot=th on kept spans
  dynamic_sampling:
    rules:
      - name: default
        sampler:
          type: ema_dynamic
          goal_sampling_percentage: 10
          key_attributes: ["service.name"]

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [probabilistic_sampler, dynamic_sampling]
      exporters: [otlphttp]

The stricter stage wins, and the surviving ot=th always reflects the effective end-to-end probability:

With the 10% goal above, roughly 10% of the original traffic survives (not 10% of the upstream sampler’s 50%), and every kept span carries the 10% threshold, so adjusted counts reconstruct the original volume.
If the rule were looser than upstream (e.g. always_sample), all arriving spans are kept and retain the upstream 50% threshold, so adjusted counts remain honest.

Accuracy under non-uniform upstream sampling

The equalizing composition above is exact when upstream sampling is uniform across the keys the rule’s adaptive sampler uses (key_attributes). If upstream head-samples different classes of traffic at different rates and those classes overlap with the tail sampler’s keys, the adaptive samplers observe a population that under-represents heavily-downsampled keys and can misjudge their per-key rate. Improving accuracy in that case requires per-key upstream tracking in the sampler and is tracked as follow-up work in #49517. Under uniform upstream sampling (the common case, e.g. an SDK TraceIdRatioBased sampler) the rates are exact.

Grouping unrelated traces with a shared `ot=rv`

Because the sampling decision is deterministic against the 56-bit randomness value, an upstream producer that sets the same ot=rv on multiple otherwise-unrelated traces will get the same sampling decision for all of them at the same threshold. This is useful when a set of traces should be sampled together as a group, for example:

An LLM agent producing multiple traces for turns in a single conversation, all stamped with a conversation.id-derived ot=rv.
A browser SDK producing multiple traces during one user session, all stamped with a session.id-derived ot=rv.
A batch job producing one trace per task, all stamped with the batch’s job ID.

The ot=rv value is a 56-bit number, so a stable hash of the entity ID truncated to 56 bits is a suitable derivation. The processor itself does not compute ot=rv from arbitrary attributes: the producer or an earlier processor is expected to set it. Whatever ot=rv is present when the accumulated trace arrives will be used for the decision and preserved on the emitted spans.

Multi-instance deployment considerations

Under the standard two-tier deployment pattern (loadbalancing exporter routing traces to a downstream tier running this processor), how traces are routed interacts with rv-based grouping in two useful ways:

Routing by trace ID (default). Traces in the same rv-group land on different collector instances because their trace IDs are unrelated. Sampling decisions remain correct without any coordination between collectors: because our decision is deterministic against the shared rv, every instance reaches the same keep/drop outcome for a given rv. The trade-off is that the adaptive samplers (ema_dynamic, ema_throughput, windowed_throughput) each see only a slice of the group’s traffic, so per-key rate calculations converge more slowly than if the whole group were visible to one instance.
Routing by a group-carrying attribute. If the producer sets both a shared conversation.id (or similar) and the derived ot=rv, loadbalancing can be configured with routing_key: attributes naming that attribute so all traces for one group land on the same collector. Adaptive-sampler learning is coherent across the group at the cost of load distribution: heavy-tailed group sizes (one active conversation among many quiet ones) skew load onto specific instances, and num_traces on the busy instance must be sized for the largest concurrently-pending group or traces_evicted will start climbing. Group size is upstream-controlled, so an unexpected traffic spike within one group shifts the skew unpredictably.

Route-by-trace-ID is the safe default (uniform load, correct decisions, coarser sampler learning). Route-by-attribute is worth reaching for when coherent adaptive learning across a group matters and num_traces can be sized conservatively. Future work on shared trace context across collector instances (tracked under “Cross-instance shared state” in #49311) would remove this trade-off: with a shared backing store the adaptive samplers can learn from the whole group regardless of which instance decides any individual trace, so uniform-load routing (route by trace ID) no longer costs sampler learning coherence.

Metrics

Metric	Type	Labels	Description
`otelcol_processor_dynamic_sampling_traces_active`	Gauge		Traces currently in the accumulation buffer.
`otelcol_processor_dynamic_sampling_traces_sampled`	Counter	`rule`	Traces kept, attributed to the rule that selected them.
`otelcol_processor_dynamic_sampling_traces_dropped`	Counter	`rule`	Traces dropped, attributed to the rule that selected them.
`otelcol_processor_dynamic_sampling_decision_sample_rate`	Histogram	`rule`	Distribution of effective sample rates produced per rule.
`otelcol_processor_dynamic_sampling_decision_triggers`	Counter	`trigger`	Number of trace decisions made, labelled by which event triggered them (`root_span`, `trace_timeout`, `eviction`, `shutdown`).
`otelcol_processor_dynamic_sampling_traces_evicted`	Counter		Traces evicted from the buffer under pressure. Each still receives a decision per the eviction policy.
`otelcol_processor_dynamic_sampling_incoming_tracestate_unparseable`	Counter		Spans whose incoming W3C tracestate could not be parsed while applying the sampling threshold.
`otelcol_processor_dynamic_sampling_ottl_eval_errors`	Counter	`rule`	OTTL condition evaluation errors, labelled by the rule the condition belongs to.

The rule label carries the matched rule’s name from the config. Values prefixed with _ are processor-owned sentinels rather than user rules: _unmatched (dropped with no matching rule), _eviction (decided by the probabilistic eviction policy), and _root_span_condition (on ottl_eval_errors, when the root-span condition itself fails to evaluate). Config validation rejects user rule names starting with _, so the two can never collide. These labels and the trigger values above are part of the component’s telemetry contract; build dashboards on them freely.

Output attributes

Every span in a sampled trace is annotated with:

Attribute	Type	Example	Description
`otelcol.processor.dynamic_sampling.rule`	string	`keep-errors`	Name of the rule that selected this trace.

The sample rate is encoded in W3C TraceState as ot=th:<hex> per the OTel consistent probability sampling spec. The spanmetrics connector (enable_metrics_sampling_method: true) reads this field to produce correctly weighted R.E.D metrics from sampled data.

Future work

Shared-storage backed scaling for single-tier deployments
Rule and sampler hot-reload
Span-count decision trigger (SpanLimit parity)
Stress-relief style overload activation
Correlated log sampling

Last generated: 2026-08-17

​Dynamicsampling Processor

​Supported Telemetry

​Overview

​How it works

​Configuration

​Root-span detection

​Rules

​How multiple conditions combine

​Condition examples

​Samplers

​deterministic

​ema_dynamic

​ema_throughput

​windowed_throughput

​Sampling keys

​Worked examples

​Error retention with an adaptive default

​Throughput-bounded fleet

​Decision cache

​Buffer overflow and eviction policy

​Deployment considerations

​Known limitations

​Relationship to processor/tail_sampling

​Decision semantics

​No rate or threshold in the policy contract

​sample_on_first_match would become correctness-load-bearing for one policy type only

​Rule attribution

​Relationship to PR #48865

​Interoperability with upstream sampling

​Worked example: composing with a head sampler

​Accuracy under non-uniform upstream sampling

​Grouping unrelated traces with a shared ot=rv

​Multi-instance deployment considerations

​Metrics

​Output attributes

​Future work