> ## Documentation Index
> Fetch the complete documentation index at: https://otel.fyi/llms.txt
> Use this file to discover all available pages before exploring further.

# Adaptivetailsampling

> OpenTelemetry processor for Adaptivetailsampling

# Adaptivetailsampling Processor

![Status](https://img.shields.io/badge/status-development-orange)

**Maintainers:** [@MikeGoldsmith](https://github.com/MikeGoldsmith), [@VinozzZ](https://github.com/VinozzZ), [@jmacd](https://github.com/jmacd)

**Source:** [opentelemetry-collector-contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/adaptivetailsamplingprocessor)

## Supported Telemetry

![Traces](https://img.shields.io/badge/traces-development-orange)

## Overview

The Adaptive Tail Sampling Processor performs adaptive tail-based trace sampling using first-match rules-based routing to adaptive samplers. Each sampler produces a known sample rate which is encoded as `ot=th` in W3C TraceState for correct downstream metric weighting.

## How it works

1. Spans are accumulated in memory, grouped by trace ID.
2. A trace becomes ready for evaluation when **any** of three triggers fires (whichever comes first):
   * a root span arrives (any span with an empty `ParentSpanID`),
   * `trace_timeout` elapses since the first span of the trace arrived. This timer is set on first-seen and is never extended by subsequent spans, ensuring a predictable upper bound on buffer occupancy, or
   * the trace accumulates `span_limit` spans (10000 by default; 0 disables). This bounds the memory a single giant trace can hold; the decision is made over the spans buffered so far, and spans arriving afterwards are stamped from the decision cache instead of being buffered.
3. After the trigger fires, the processor pauses for `decision_delay` to let in-flight straggler spans land. The root-span and trace-timeout triggers share the same delay; the span-limit trigger decides immediately, since the trace is at its memory peak and waiting would let it keep growing.
4. Rules are then evaluated in order against the accumulated trace. The first rule whose conditions all match selects the sampler; once a rule is selected, its sampler's keep/drop decision is final and no later rules are considered. A rule with no conditions is a catch-all.
5. The matched sampler produces a sample rate (1-in-N). Samplers only ever produce rates; no sampler makes a keep/drop decision itself.
6. The processor converts the rate to a threshold (a rate of N becomes the threshold encoding probability 1/N) and makes the keep/drop decision by comparing that threshold against the trace's randomness (`ot=rv` when present, otherwise derived from the trace ID), using the [OTel consistent probability sampling](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/) algorithm.

> \[!NOTE]
> The adaptive samplers come from [dynsampler-go](https://github.com/honeycombio/dynsampler-go), which the processor uses **only** to compute rates. dynsampler-go's own samplers can make keep/drop decisions internally (its `DeterministicSampler` applies its own hash-based check, for example), but this processor never uses that path: the rate-to-threshold conversion and the randomness comparison in step 6 are the single decision mechanism for every sampler type, including this processor's `probabilistic` type. This is what makes decisions reproducible for a given trace and correctly weighted downstream via `ot=th`.

7. Sampled traces are forwarded with annotations on every span:
   * `otelcol.processor.adaptive_tail_sampling.rule`: the name of the matched rule
   * `otelcol.processor.adaptive_tail_sampling.trigger`: which event triggered the decision (see [Output attributes](#output-attributes))
   * W3C TraceState `ot=th:<hex>`: the threshold encoding the effective sample rate
8. The decision (sampled or dropped) is recorded in a per-trace LRU cache so that any late-arriving spans for the same trace ID are handled consistently with the original decision (see [Decision cache](#decision-cache) below).

## Configuration

```yaml theme={null}
processors:
  adaptive_tail_sampling:
    # Upper bound on time a trace can sit in the buffer before its decision is
    # forced. Acts as a safety net for traces that never see a root span.
    trace_timeout: 30s

    # Pause between a triggering event (root span arrival or trace timeout) and
    # the actual decision evaluation. Allows in-flight straggler spans to land
    # before the trace is decided.
    decision_delay: 2s

    # Maximum number of traces held in the in-memory buffer.
    num_traces: 50000

    # Maximum spans a single trace may accumulate before it is decided
    # immediately over the spans buffered so far. Bounds the memory one giant
    # trace can hold; num_traces and eviction only bound the trace count.
    # Defaults to 10000; 0 disables the cap.
    span_limit: 10000

    decision_cache:
      sampled_cache_size: 10000      # 0 disables the sampled cache
      non_sampled_cache_size: 10000  # 0 disables the not-sampled cache

    rules:
      # First-match: the first rule whose conditions all match selects the sampler.
      - name: keep-errors
        conditions:
          - "status.code == 2"
        sampler:
          type: always_sample

      - name: payment-service
        conditions:
          - 'resource.attributes["service.name"] == "payment"'
        sampler:
          type: adaptive_percentage
          goal_percentage: 5
          fingerprint_attributes:
            - span.attributes["http.method"]
            - span.attributes["http.route"]
          adjustment_interval: 15s
          weight: 0.5

      # Catch-all: no conditions, always matches.
      - name: default
        sampler:
          type: adaptive_percentage
          goal_percentage: 20
          fingerprint_attributes:
            - resource.attributes["service.name"]
            - span.attributes["http.status_code"]

    # Optional. OTTL boolean expression evaluated per span. When it returns true
    # for any span in the trace, the trace transitions from accumulation to the
    # decision-delay phase without waiting for trace_timeout. Defaults to
    # IsRootSpan() when unset.
    # root_span_condition: 'IsRootSpan() or span.attributes["otelcol.adaptive_tail_sampling.root_span"] == true'
```

### Root-span detection

The processor moves a trace out of accumulation and into the decision-delay phase as soon as it sees a span it considers a "root". By default that means `IsRootSpan()`: any span whose W3C `ParentSpanID` is empty. This works for producer-side head-sampled traces where the true root always lands at the same collector, but it stalls when:

* Only the server side of a cross-process trace reaches this collector, so no observed span has an empty parent.
* The operator wants to trigger on a producer-supplied hint (e.g. a message-broker consumer or batch-job entry point) rather than on the transport-level root.

`root_span_condition` lets the operator override the trigger with any OTTL boolean expression evaluated in the `ottlspan` context. The first span for which it returns true starts the decision-delay timer.

Producer hint (fires on any real root, or on any span the producer explicitly tagged):

```yaml theme={null}
root_span_condition: 'IsRootSpan() or span.attributes["otelcol.adaptive_tail_sampling.root_span"] == true'
```

Cross-process server (accept the gateway's server span as a trigger even though it has a parent):

```yaml theme={null}
root_span_condition: 'IsRootSpan() or (span.kind == SPAN_KIND_SERVER and resource.attributes["service.name"] == "gateway")'
```

Message consumer (trigger on the consumer side of a queue rather than waiting for `trace_timeout`):

```yaml theme={null}
root_span_condition: 'IsRootSpan() or (span.kind == SPAN_KIND_CONSUMER and IsMatch(span.name, "^receive "))'
```

Evaluation errors are counted on `processor_adaptive_tail_sampling_ottl_eval_errors` under the sentinel `rule="_root_span_condition"` label so they show up separately from rule-condition errors. A span whose evaluation errored is treated as non-matching, so a broken expression can only delay the trigger to `trace_timeout`, never fire it prematurely.

### Rules

Rules are evaluated in order; the first whose conditions all match selects the sampler. A rule with no conditions is a catch-all. Once a rule is selected, its sampler's decision is final: a drop stays dropped, the trace is not handed to any later rule.

Each rule's `name` must be unique. It is recorded on the decision metrics (the `rule` attribute) and stamped on sampled spans, so pick names you want to see on dashboards. Names starting with `_` are rejected at config validation: that prefix is reserved for processor-internal decision labels (such as `_eviction`), so user rules can never collide with them.

> \[!WARNING]
> A rule with no conditions (a catch-all) placed before another rule consumes every trace and renders the later rules unreachable. The processor logs a warning at startup when it detects this configuration so it shows up in collector logs.

Conditions are [OTTL](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl) boolean expressions evaluated in the [`ottlspan`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl/contexts/ottlspan) context, which gives access to the span, its resource, and its instrumentation scope. Path expressions should be qualified with the context they refer to (`span.attributes["k"]`, `resource.attributes["k"]`, `span.status.code`, and so on); unqualified paths are valid OTTL and resolve against the span context. We recommend qualifying every path, all examples in this README do, and it keeps conditions visually consistent with fingerprint selectors, which require a scope. See [OTTL Boolean Expressions](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/LANGUAGE.md#boolean-expressions) for the full grammar (including `and`, `or`, and parentheses), and the [ottlspan paths reference](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottlspan/README.md) for the list of accessible fields.

The intended pattern is specific-conditions rules first, catch-all last:

```yaml theme={null}
rules:
  - name: keep-errors
    conditions:
      - span.status.code == STATUS_CODE_ERROR
    sampler:
      type: always_sample
  - name: default              # catch-all
    sampler:
      type: adaptive_percentage
      goal_percentage: 10
      fingerprint_attributes:
        - resource.attributes["service.name"]
```

With the order above, error traces are always kept and every other trace is decided by `adaptive_percentage`. Flipping the order so `default` comes first would mean `default` swallows every trace (including errors) and `keep-errors` is never reached, which is what the startup warning flags.

#### How multiple conditions combine

Each rule can carry multiple OTTL conditions. The `match` field controls how they are combined against the spans of the accumulated trace:

* `match: any_span` (default): each condition must be satisfied by *some* span in the trace, but not necessarily the same span across conditions. Reads as "the trace has these characteristics."
* `match: same_span`: some single span in the trace must satisfy *all* conditions at once. Reads as "there is a span with these characteristics."

`match` is not permitted on catch-all rules (rules with no conditions).

Example: keep a trace only when a single span is both an error and from the payment service:

```yaml theme={null}
- name: payment-errors
  match: same_span
  conditions:
    - span.status.code == STATUS_CODE_ERROR
    - resource.attributes["service.name"] == "payment"
  sampler:
    type: always_sample
```

Example (default `any_span`): keep any trace that touched the payment service *and* saw an error somewhere:

```yaml theme={null}
- name: payment-involved-errors
  conditions:
    - span.status.code == STATUS_CODE_ERROR
    - resource.attributes["service.name"] == "payment"
  sampler:
    type: always_sample
```

If an OTTL condition raises an evaluation error at runtime (for example, a path does not exist on a specific span) the condition is treated as false for that span and the `otelcol_processor_adaptive_tail_sampling_ottl_eval_errors` counter is incremented, labelled by the rule name. Other spans and conditions continue to evaluate normally.

#### Condition examples

Keep every trace where an HTTP server span returned 5xx, using OTTL's inline `and`:

```yaml theme={null}
- name: http-5xx
  match: same_span
  conditions:
    - span.kind == SPAN_KIND_SERVER and span.attributes["http.response.status_code"] >= 500
  sampler:
    type: always_sample
```

Keep every trace that includes a root span from a specific service, using OTTL's `IsRootSpan()` helper:

```yaml theme={null}
- name: root-checkout
  match: same_span
  conditions:
    - IsRootSpan() and resource.attributes["service.name"] == "checkout"
  sampler:
    type: always_sample
```

Keep traces from either the checkout or the billing service, at a modest adaptive rate:

```yaml theme={null}
- name: high-value-services
  conditions:
    - resource.attributes["service.name"] == "checkout" or resource.attributes["service.name"] == "billing"
  sampler:
    type: adaptive_percentage
    goal_percentage: 25
    fingerprint_attributes:
      - resource.attributes["service.name"]
      - span.attributes["http.route"]
```

Match on span name prefix using OTTL's `IsMatch` regex helper:

```yaml theme={null}
- name: api-calls
  match: same_span
  conditions:
    - IsMatch(span.name, "^GET /api/")
  sampler:
    type: probabilistic
    sampling_percentage: 5
```

For the full set of comparison operators, string functions, and converter helpers available in conditions, see [OTTL Functions](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/ottlfuncs/README.md).

### Samplers

Each type names a distinct intent:

| Type                  | Behaviour                                                                                    |
| --------------------- | -------------------------------------------------------------------------------------------- |
| `always_sample`       | Keep every matching trace (sample rate 1).                                                   |
| `probabilistic`       | Keep a fixed fraction of traces, the same for all traffic (`sampling_percentage`, (0, 100]). |
| `adaptive_percentage` | Adapt per key toward a target percentage of traffic (`goal_percentage`, (0, 100]).           |
| `adaptive_throughput` | Adapt per key toward a volume budget in spans per second (`goal_throughput`, > 0).           |

Decisions are always made per trace; volume is measured in spans, so a kept
trace counts all of its spans against a throughput budget, and a percentage
goal is measured over span volume (equal to the percentage of traces when
trace sizes are uniform).

The `adaptive_*` samplers are backed by
[`dynsampler-go`](https://github.com/honeycombio/dynsampler-go) and accept an
optional `algorithm` selecting how per-key rates are computed: `ema` (the
default) smooths traffic with an exponential moving average, tuned with
`adjustment_interval` and `weight`; `windowed` (supported by
`adaptive_throughput` only) computes rates over a sliding window, reacting
faster to traffic shifts at the cost of more spike sensitivity, tuned with
`update_frequency` and `lookback_frequency`.

Migrating from Refinery: `DeterministicSampler` -> `probabilistic` (the same
hash-consistent fixed fraction), `EMADynamicSampler` -> `adaptive_percentage`
(note Refinery's `GoalSampleRate: N` means keep 1-in-N, so `GoalSampleRate: 5`
becomes `goal_percentage: 20`), `EMAThroughputSampler` -> `adaptive_throughput`,
`WindowedThroughputSampler` -> `adaptive_throughput` with `algorithm: windowed`.

#### `probabilistic`

The inline equivalent of the `probabilistic_sampler` processor.

```yaml theme={null}
sampler:
  type: probabilistic
  sampling_percentage: 10   # keep 10% of traces
```

#### `adaptive_percentage`

Adapts the sample rate per traffic key over time, keeping a target average
percentage across all keys: rare keys survive, chatty keys are sampled
aggressively.

```yaml theme={null}
sampler:
  type: adaptive_percentage
  goal_percentage: 10                     # target % across all keys
  fingerprint_attributes:
    - resource.attributes["service.name"]
    - span.attributes["http.status_code"]
  adjustment_interval: 15s                # how often the ema recalculates
  weight: 0.5                             # ema weighting factor in [0, 1); 0 or omitted = 0.5
  max_keys: 500                           # 0 = unlimited
```

#### `adaptive_throughput`

Adjusts rates per key to hit a sustained volume budget in spans per second.

```yaml theme={null}
sampler:
  type: adaptive_throughput
  goal_throughput: 100                    # target spans/sec per instance, across all keys
  fingerprint_attributes:
    - resource.attributes["service.name"]
    - span.attributes["http.status_code"]
  adjustment_interval: 15s
  weight: 0.5
  max_keys: 500
```

> \[!IMPORTANT]
> `goal_throughput` is enforced **per collector instance**. Each instance targets the goal against the traffic it sees, so a fleet of N instances emits up to N times the configured throughput. Divide the backend budget by the instance count when sizing this value. See [Deployment considerations](#deployment-considerations).

With `algorithm: windowed`, rate recalculation (`update_frequency`) is
decoupled from the historical window used for the calculation
(`lookback_frequency`):

```yaml theme={null}
sampler:
  type: adaptive_throughput
  algorithm: windowed
  goal_throughput: 100
  fingerprint_attributes:
    - resource.attributes["service.name"]
    - span.attributes["http.status_code"]
  update_frequency: 1s                    # how often rates recalculate; 0 or omitted = 1s
  lookback_frequency: 30s                 # historical window; floored to a multiple of update_frequency
  max_keys: 500
```

### Fingerprints

The `fingerprint_attributes` field names the attributes that identify what kind of trace this is for sampling purposes. Each distinct fingerprint value gets its own adaptive sample rate, so choose attributes that classify traffic (route, status code, method, service) rather than identify individual requests (user IDs, request IDs, raw URLs), which would give every trace its own key and defeat the adaptation.

Each entry is a scoped attribute selector of the form `<scope>.attributes["<name>"]`:

| Scope       | Reads from                                                        |
| ----------- | ----------------------------------------------------------------- |
| `resource.` | each resource's attributes                                        |
| `scope.`    | each instrumentation scope's attributes                           |
| `span.`     | every span's attributes                                           |
| `root.`     | the spans matching the configured `root_span_condition`           |
| `any.`      | the union of resource, instrumentation scope, and span attributes |

The `resource.`, `scope.`, and `span.` prefixes match OTTL's span-context path names, so conditions and fingerprint entries share one spelling; `root.` and `any.` are trace-level scopes OTTL cannot express (fingerprints are built from the whole trace, while OTTL evaluates one span at a time). Note that fingerprint entries are selectors, not OTTL expressions. The two concepts have distinct jobs throughout the config: OTTL conditions appear wherever a single span is evaluated (`conditions`, `root_span_condition`), and selectors appear wherever a trace-level value is collected. Qualifying every path in conditions keeps the two styles identical in practice.

```yaml theme={null}
fingerprint_attributes:
  - resource.attributes["service.name"]
  - span.attributes["http.route"]
  - root.attributes["http.status_code"]
```

Every scope collects **all** distinct matching values, there is no first-match or precedence: `any.` is simply the widest search, and a value present at several origins appears once. The fingerprint for a trace is built by sorting the distinct values each selector matched and joining them with `,` within each entry, then joining the entries with the `•` separator. A trace whose spans carry several values for one selector keys as the combination (e.g. `checkout,billing•/api`), which is worth knowing when debugging unexpectedly high key cardinality. Selectors that match nothing are replaced with `<missing>`.

Extraction cost scales with the scope: `resource.` and `scope.` are independent of trace size, while `span.`, `any.`, and `root.` walk every span of the trace at decision time. `root.` additionally evaluates the root-span condition per span, which is cheap for the default condition but costs an OTTL evaluation per span for custom ones.

## Worked examples

Two complete configurations for the most common deployment shapes.

### Error retention with an adaptive default

Keep every error trace, and let an adaptive sampler settle the rest at a target percentage per traffic class. This is the right starting shape for most single-instance deployments: errors are never lost, and the default rule adapts as traffic mix shifts.

```yaml theme={null}
processors:
  adaptive_tail_sampling:
    trace_timeout: 30s
    decision_delay: 2s
    num_traces: 50000
    decision_cache:
      sampled_cache_size: 50000
      non_sampled_cache_size: 100000
    rules:
      # Errors are always kept, evaluated first.
      - name: keep-errors
        conditions:
          - span.status.code == STATUS_CODE_ERROR
        sampler:
          type: always_sample
      # Everything else settles at ~10% per (service, route) class, so
      # low-traffic routes stay visible while hot routes are downsampled.
      - name: default
        sampler:
          type: adaptive_percentage
          goal_percentage: 10
          fingerprint_attributes:
            - resource.attributes["service.name"]
            - span.attributes["http.route"]
          adjustment_interval: 15s
          weight: 0.5
```

Sizing guidance: `num_traces` bounds memory and should cover the number of traces that start within one `trace_timeout` window at peak. The decision caches should be a small multiple of that, since they only store trace IDs and outcomes; undersizing them turns late spans of decided traces back into new partial traces. `num_traces` bounds how many traces are buffered, not how large any one of them grows; `span_limit` (10000 by default) covers that dimension. Keep it well above your largest legitimate trace so a single runaway trace cannot exhaust memory while normal traces are never truncated.

### Throughput-bounded fleet

Cap the spans-per-second this collector emits regardless of incoming volume, useful when the downstream backend is provisioned for a fixed ingest rate. The throughput sampler continuously adjusts per-key rates to hold the total near the goal.

```yaml theme={null}
processors:
  adaptive_tail_sampling:
    trace_timeout: 30s
    decision_delay: 2s
    num_traces: 100000
    decision_cache:
      sampled_cache_size: 100000
      non_sampled_cache_size: 200000
    # Under sustained overload, shed evicted traces in constant time instead
    # of paying rule evaluation for every one (see Buffer overflow and
    # eviction policy below).
    eviction:
      policy: probabilistic
      sampling_percentage: 10
    rules:
      - name: throughput-cap
        sampler:
          type: adaptive_throughput
          algorithm: windowed
          goal_throughput: 1000
          fingerprint_attributes:
            - resource.attributes["service.name"]
          update_frequency: 1s
          lookback_frequency: 30s
```

The goal is enforced per collector instance: a fleet of N instances emits up to N times the configured throughput, so divide the backend budget by the instance count. Keying by `service.name` means each service's share adapts to its share of total traffic rather than being fixed.

## Decision cache

When a trace's decision is finalised, the trace ID and outcome are recorded in
one of two LRU caches. Any later spans that arrive for that trace ID short-circuit
the accumulation path and are handled consistently with the original decision:

* **Sampled cache** (`decision_cache.sampled_cache_size`): late spans are forwarded
  immediately, stamped with the same rule attribute and `ot=th` TraceState as the
  original batch.
* **Not-sampled cache** (`decision_cache.non_sampled_cache_size`): late spans are
  silently dropped.

Setting a cache size to `0` disables that side of the cache. Late spans for a
decision class whose cache is disabled fall through to the normal pending-trace
path and may produce a second (possibly inconsistent) decision; with both caches
disabled, every late span falls through. Operators tune the cache sizes against
observed late-span volume.

### Buffer overflow and eviction policy

The in-memory accumulation buffer is sized by `num_traces`. When the buffer is
full and spans for brand-new traces arrive, the processor evicts the **oldest**
pending traces to make room (the buffer may transiently exceed `num_traces` by
the number of new traces in a single incoming batch). Every evicted trace
receives a real sampling decision immediately, with the spans seen so far and
no `decision_delay`; the decision is recorded in the decision cache so
late-arriving spans are handled consistently, and kept traces carry correct
`ot=th` annotations. How that decision is made is configurable:

```yaml theme={null}
eviction:
  policy: evaluate            # evaluate (default) | probabilistic
  sampling_percentage: 10     # required for probabilistic
```

* `evaluate` (default): the evicted trace runs through the normal rules and
  sampler path. Your rules keep working under pressure (e.g. a keep-errors
  rule still keeps error traces), at the cost of OTTL evaluation over every
  span of every evicted trace.
* `probabilistic`: rule evaluation is skipped and the trace is decided by
  comparing a threshold derived from `sampling_percentage` against the trace's
  randomness. Constant-time work per evicted trace regardless of trace size or
  rule count, so an overloaded instance sheds load instead of amplifying it.
  Kept traces are stamped with the corresponding `ot=th` (and the sentinel
  rule attribution `_eviction`), so downstream weighting stays accurate.
  Recommended for high-throughput deployments where eviction indicates
  genuine overload.

Note the decision may be made on an incomplete trace in both modes: spans
still in flight when the trace is evicted are treated as late spans and follow
the recorded decision. The decision cache (`decision_cache.*`) is a separate
structure recording *completed* decisions; it does not protect pending traces
from eviction.

Operators sizing the processor should watch `traces_evicted` (every eviction)
and `decision_triggers{trigger="eviction"}` and increase `num_traces` if they
are non-zero in steady state.

## Deployment considerations

The processor accumulates spans in memory, so all spans of a given trace must reach the same processor instance. Multi-instance deployments use the same two-tier pattern as the `tail_sampling` processor:

```
SDKs → Collectors (loadbalancing exporter, hash by traceID)
         → Collectors (adaptive_tail_sampling processor)
           → Backend
```

Each processor instance runs its samplers independently against the traffic it sees; there is no coordination between instances. For `adaptive_throughput` (with either algorithm) this means `goal_throughput` is a **per-instance** target: a fleet of N instances emits up to N times the configured goal, so divide the backend's ingest budget by the instance count when sizing it. The percentage-based samplers (`adaptive_percentage`, `probabilistic`) are unaffected, since a target percentage composes across instances. Automatic cluster-size awareness for the throughput goal is not currently implemented.

## Known limitations

* **Decisions are per-instance.** All spans of a trace must reach the same
  instance; scale out with the two-tier `loadbalancing` pattern above. Shared
  storage for a single-tier deployment is future work.
* **The buffer is bounded by trace count, not bytes.** Steady-state memory is
  approximately span rate x buffer window (`decision_delay` after the decision
  triggers, up to `trace_timeout`) x average span size, plus the decision
  caches. Trace shape barely matters: per-trace bookkeeping is small, so many
  small traces cost slightly more than a few giant ones at the same span rate.
  When `num_traces` is exceeded the oldest trace is evicted with a real
  decision (see the eviction policy section).
* **Shutdown drains rather than discards.** Pending traces are decided with
  the spans seen so far and kept traces are forwarded, so a clean restart or
  rollout does not silently lose the buffer. A crash still loses it.
* **Rules are fixed at startup.** Changing rules means a restart (which
  drains, see above). Hot reload is future work.
* **Traces only.** Rule conditions use the OTTL span context, and the
  configuration is trace-scoped; correlated log sampling is future work and
  will add log-specific configuration alongside the existing fields.

## Relationship to `processor/tail_sampling`

This processor sits next to `tail_sampling` rather than extending it. The two are close in shape (both buffer traces and decide once a trigger fires) but use different evaluation models, and `tail_sampling`'s current model has several mechanical features that a rate-bearing sampler does not fit cleanly into.

### Decision semantics

In `tail_sampling`'s [policy loop](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/processor.go#L800-L817) only two outcomes short-circuit evaluation: any policy returning `Dropped`, and (when `sample_on_first_match` is enabled) the first policy to return `Sampled`. Every other outcome, including `NotSampled`, lets the loop continue to subsequent policies.

All common "this trace did not pass my check" votes from existing policies (probabilistic, status\_code, rate\_limiting, and\_policy, etc.) return `NotSampled`, never `Dropped`. `Dropped` is reserved for the explicit [`drop` policy](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/internal/sampling/drop.go#L46), and the [final-decision composition](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/processor.go#L821-L836) gives `Dropped` precedence over `Sampled`.

A rate-bearing sampler dropped into this model has to pick a return value for a probabilistically-dropped trace, and both options break correctness:

* Returning `NotSampled` matches the existing convention but does not stop the loop. A later policy voting `Sampled` would still cause the trace to be kept, and the rate-bearing sampler's view of what it controlled diverges from reality. Its rate calculations drift over time.
* Returning `Dropped` stops the loop, but it also wins precedence over every `Sampled` vote in the composition step. An operator pairing a rate-bearing sampler with an explicit `keep-errors` policy would expect errors to always win; instead the rate-bearing sampler's probabilistic drop would override the keep, inverting the configured intent.

### No rate or threshold in the policy contract

The [`Evaluator` interface](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/pkg/samplingpolicy/samplingpolicy.go#L78-L85) returns only a decision enum. There is no way for a policy to communicate the sample rate or threshold it applied, which means `ot=th` cannot be emitted from inside `tail_sampling` without expanding the contract. The processor has roughly twenty existing policies; widening the interface would touch all of them.

### `sample_on_first_match` would become correctness-load-bearing for one policy type only

Today `sample_on_first_match` is an opt-in optimization. Adding rate-bearing samplers as policies would make it mandatory for configurations using them; without it, the OR composition above produces drift. Existing `tail_sampling` users adding a rate-bearing policy without flipping the flag would silently get wrong rates. Validation cannot easily reject this because the flag is fine in isolation, and there is no current marker on a policy type that says "this policy requires first-match for correctness."

### Rule attribution

This processor records the matched rule name on every span in a sampled trace, which is only meaningful under first-match semantics. Under `tail_sampling`'s multi-policy OR model multiple policies can vote `Sampled` and there is no defined "winner" to attribute the trace to.

### Relationship to [PR #48865](https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/48865)

In-flight work on adding tracestate handling to `tail_sampling`'s probabilistic policy is input-driven: it reads SDK-supplied `ot=th` to adjust the tail probabilistic threshold (equalizing mode). This processor is output-driven: it computes a tail-stage rate and emits `ot=th`. The two address different problems and can ship independently.

For these reasons `adaptive_tail_sampling` is a separate processor. The `tail_sampling` users retain the existing multi-policy model unchanged, and `adaptive_tail_sampling` keeps a single evaluation model (first-match with rate-bearing samplers) end to end.

## Interoperability with upstream sampling

The processor honours any incoming `ot=th` (sampling threshold) and `ot=rv` (explicit randomness) already set on incoming spans by an upstream sampler, such as an SDK head sampler or a probabilistic collector processor:

* **Randomness.** If an incoming span carries `ot=rv`, that value is used to make the sampling decision. Otherwise the last 7 bytes of the trace ID are used, per the consistent probability spec.
* **Population-relative rate (equalizing).** The rule's rate `N` is interpreted as the operator's target for the original population: "keep 1-in-N of all traces before any sampling." The effective absolute keep probability is `min(P_upstream, 1/N)`. This is the same composition mode as `equalizing` in [`processor/probabilisticsamplerprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/probabilisticsamplerprocessor#equalizing).
* **Threshold monotonicity.** If a span already carries an `ot=th` stricter than what this processor would emit, the incoming value is preserved. This matches the consistent probability spec: a downstream stage may raise a threshold but never lower it.

### Worked example: composing with a head sampler

```yaml theme={null}
processors:
  probabilistic_sampler:
    sampling_percentage: 50
    mode: equalizing # writes ot=th on kept spans
  adaptive_tail_sampling:
    rules:
      - name: default
        sampler:
          type: adaptive_percentage
          goal_percentage: 10
          fingerprint_attributes:
            - resource.attributes["service.name"]

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [probabilistic_sampler, adaptive_tail_sampling]
      exporters: [otlphttp]
```

The stricter stage wins, and the surviving `ot=th` always reflects the
effective end-to-end probability:

* With the 10% goal above, roughly 10% of the *original* traffic survives (not
  10% of the upstream sampler's 50%), and every kept span carries the 10%
  threshold, so adjusted counts reconstruct the original volume.
* If the rule were looser than upstream (e.g. `always_sample`), all arriving
  spans are kept and retain the upstream 50% threshold, so adjusted counts
  remain honest.

### Accuracy under non-uniform upstream sampling

The equalizing composition above is exact when upstream sampling is uniform across the keys the rule's adaptive sampler uses (`fingerprint_attributes`). If upstream head-samples different classes of traffic at different rates and those classes overlap with the tail sampler's keys, the adaptive samplers observe a population that under-represents heavily-downsampled keys and can misjudge their per-key rate. Improving accuracy in that case requires per-key upstream tracking in the sampler and is tracked as follow-up work in [#49517](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/49517). Under uniform upstream sampling (the common case, e.g. an SDK `TraceIdRatioBased` sampler) the rates are exact.

### Grouping unrelated traces with a shared `ot=rv`

Because the sampling decision is deterministic against the 56-bit randomness value, an upstream producer that sets the same `ot=rv` on multiple otherwise-unrelated traces will get the same sampling decision for all of them at the same threshold. This is useful when a set of traces should be sampled together as a group, for example:

* An LLM agent producing multiple traces for turns in a single conversation, all stamped with a `conversation.id`-derived `ot=rv`.
* A browser SDK producing multiple traces during one user session, all stamped with a `session.id`-derived `ot=rv`.
* A batch job producing one trace per task, all stamped with the batch's job ID.

The `ot=rv` value is a 56-bit number, so a stable hash of the entity ID truncated to 56 bits is a suitable derivation. The processor itself does not compute `ot=rv` from arbitrary attributes: the producer or an earlier processor is expected to set it. Whatever `ot=rv` is present when the accumulated trace arrives will be used for the decision and preserved on the emitted spans.

#### Multi-instance deployment considerations

Under the standard two-tier deployment pattern (`loadbalancing` exporter routing traces to a downstream tier running this processor), how traces are routed interacts with rv-based grouping in two useful ways:

* **Routing by trace ID (default).** Traces in the same rv-group land on different collector instances because their trace IDs are unrelated. Sampling decisions remain correct without any coordination between collectors: because our decision is deterministic against the shared rv, every instance reaches the same keep/drop outcome for a given rv. The trade-off is that the adaptive samplers (`adaptive_percentage`, `adaptive_throughput`) each see only a slice of the group's traffic, so per-key rate calculations converge more slowly than if the whole group were visible to one instance.
* **Routing by a group-carrying attribute.** If the producer sets both a shared `conversation.id` (or similar) and the derived `ot=rv`, `loadbalancing` can be configured with `routing_key: attributes` naming that attribute so all traces for one group land on the same collector. Adaptive-sampler learning is coherent across the group at the cost of load distribution: heavy-tailed group sizes (one active conversation among many quiet ones) skew load onto specific instances, and `num_traces` on the busy instance must be sized for the largest concurrently-pending group or `traces_evicted` will start climbing. Group size is upstream-controlled, so an unexpected traffic spike within one group shifts the skew unpredictably.

Route-by-trace-ID is the safe default (uniform load, correct decisions, coarser sampler learning). Route-by-attribute is worth reaching for when coherent adaptive learning across a group matters and `num_traces` can be sized conservatively.

Future work on shared trace context across collector instances (tracked under "Cross-instance shared state" in [#49311](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/49311)) would remove this trade-off: with a shared backing store the adaptive samplers can learn from the whole group regardless of which instance decides any individual trace, so uniform-load routing (route by trace ID) no longer costs sampler learning coherence.

## Metrics

| Metric                                                                     | Type      | Labels    | Description                                                                                                                                                           |
| -------------------------------------------------------------------------- | --------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `otelcol_processor_adaptive_tail_sampling_traces_active`                   | Gauge     |           | Traces currently in the accumulation buffer.                                                                                                                          |
| `otelcol_processor_adaptive_tail_sampling_traces_sampled`                  | Counter   | `rule`    | Traces kept, attributed to the rule that selected them.                                                                                                               |
| `otelcol_processor_adaptive_tail_sampling_traces_dropped`                  | Counter   | `rule`    | Traces dropped, attributed to the rule that selected them.                                                                                                            |
| `otelcol_processor_adaptive_tail_sampling_decision_sample_rate`            | Histogram | `rule`    | Distribution of effective sample rates produced per rule.                                                                                                             |
| `otelcol_processor_adaptive_tail_sampling_decision_triggers`               | Counter   | `trigger` | Number of trace decisions made, labelled by which event triggered them (`root_span`, `trace_timeout`, `eviction`, `shutdown`, `span_limit`).                          |
| `otelcol_processor_adaptive_tail_sampling_trace_span_count`                | Histogram | `rule`    | Distribution of buffered span counts per trace at decision time. Useful for sizing `span_limit` against the real trace-size distribution.                             |
| `otelcol_processor_adaptive_tail_sampling_fingerprint_duration`            | Histogram | `rule`    | Time spent extracting a rule's fingerprint per decision (microseconds). A relative signal for spotting expensive fingerprints, eg wide `any.` scopes on large traces. |
| `otelcol_processor_adaptive_tail_sampling_traces_evicted`                  | Counter   |           | Traces evicted from the buffer under pressure. Each still receives a decision per the eviction policy.                                                                |
| `otelcol_processor_adaptive_tail_sampling_incoming_tracestate_unparseable` | Counter   |           | Spans whose incoming W3C tracestate could not be parsed while applying the sampling threshold.                                                                        |
| `otelcol_processor_adaptive_tail_sampling_ottl_eval_errors`                | Counter   | `rule`    | OTTL condition evaluation errors, labelled by the rule the condition belongs to.                                                                                      |

The `rule` label carries the matched rule's name from the config. Values
prefixed with `_` are processor-owned sentinels rather than user rules:
`_unmatched` (dropped with no matching rule), `_eviction` (decided by the
probabilistic eviction policy), and `_root_span_condition` (on
`ottl_eval_errors`, when the root-span condition itself fails to evaluate).
Config validation rejects user rule names starting with `_`, so the two can
never collide. These labels and the `trigger` values above are part of the
component's telemetry contract; build dashboards on them freely.

## Output attributes

Every span in a sampled trace is annotated with:

| Attribute                                              | Type   | Example            | Description                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| ------------------------------------------------------ | ------ | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `otelcol.processor.adaptive_tail_sampling.rule`        | string | `keep-errors`      | Name of the rule that selected this trace.                                                                                                                                                                                                                                                                                                                                                                                                  |
| `otelcol.processor.adaptive_tail_sampling.fingerprint` | string | `9f86d081884c7d65` | The matched rule's fingerprint, raw or hashed, when `record_fingerprint` is enabled.                                                                                                                                                                                                                                                                                                                                                        |
| `otelcol.processor.adaptive_tail_sampling.trigger`     | string | `root_span`        | What caused the decision (`root_span`, `trace_timeout`, `eviction`, `shutdown`, `span_limit`). Eviction and shutdown override the original trigger on traces they decide mid-delay, so the attribute always names the event that produced the decision, while the decision-triggers metric counts the event that first made the trace decision-eligible. Values other than `root_span` mean the decision may have seen an incomplete trace. |

### Recording the fingerprint

`record_fingerprint` (default `none`) stamps the matched rule's fingerprint on every span of a kept trace, the same way the rule name is recorded:

* `value` records the raw fingerprint (eg `checkout,billing•/api`). Long combination keys grow the sampled decision cache, which stores the recorded string for late spans.
* `hash` records the first 8 bytes of the fingerprint's SHA-256 as 16 hex characters. The size is fixed and the hash is deterministic across instances and restarts, so grouping works fleet-wide. Verify a span's hash by recomputing it from the raw fingerprint, `echo -n '<fingerprint>' | sha256sum | cut -c1-16`.

Hashing obfuscates values and fixes the attribute size. It does not protect guessable values, anyone who knows the attribute space can enumerate candidate fingerprints and hash them. Rules whose sampler has no `fingerprint_attributes` (`always_sample`, `probabilistic`) never produce the attribute, and enabling either mode adds one attribute write per span on kept traces.

The sample rate is encoded in W3C TraceState as `ot=th:<hex>` per the OTel consistent probability sampling spec. The `spanmetrics` connector (`enable_metrics_sampling_method: true`) reads this field to produce correctly weighted R.E.D metrics from sampled data.

## Future work

* Shared-storage backed scaling for single-tier deployments
* Rule and sampler hot-reload
* Span-count decision trigger (SpanLimit parity)
* Stress-relief style overload activation
* Correlated log sampling

***

*Last generated: 2026-08-31*