> ## Documentation Index
> Fetch the complete documentation index at: https://otel.fyi/llms.txt
> Use this file to discover all available pages before exploring further.

# Groupbytrace

> OpenTelemetry processor for Groupbytrace

# Groupbytrace Processor

![Status](https://img.shields.io/badge/status-beta-yellow)

**Available in:** `contrib`, `k8s`

**Maintainers:** [@iblancasa](https://github.com/iblancasa)

**Source:** [opentelemetry-collector-contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/groupbytraceprocessor)

## Supported Telemetry

![Traces](https://img.shields.io/badge/traces-beta-orange)

## Overview

This processor collects all the spans from the same trace, waiting a
pre-determined amount of time before releasing the trace to the next processor.
The expectation is that, generally, traces will be complete after the given time.

This processor should be used whenever a processor requires grouped traces to make decisions,
such as a tail-based sampler or a per-trace metrics processor. Note that [`tailsamplingprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor)
also implements a similar mechanism and can be used independently.

The batch processor shouldn't be used before this processor, as this one will
probably undo part (or much) of the work that the batch processor performs. It's
fine to have the batch processor to run right after this one, and every entry in the
batch will be a complete trace.

Please refer to [config.go](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/groupbytraceprocessor/config.go) for the config spec.

Examples:

```yaml theme={null}
processors:
  groupbytrace:
  groupbytrace/2:
    wait_duration: 10s
    num_traces: 1000
    num_workers: 2
  groupbytrace/service:
    wait_duration: 5s
    num_traces: 100000
    emit_strategy: service
```

## Configuration

Refer to [config.yaml](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/groupbytraceprocessor/testdata/config.yaml) for detailed examples on using the processor.

The `num_traces` (default=1,000,000) property tells the processor what's the maximum number of traces to keep in the internal storage. A higher `num_traces` might incur in a higher memory usage. In `emit_strategy: service` mode it bounds the number of `(trace, service)` groups awaiting release rather than the number of traces. Note that in either mode it bounds a count of traces or subtraces, not a count of spans, so memory usage also scales with how many spans each one contains.

The `wait_duration` (default=1s) property tells the processor for how long it should keep traces in the internal storage. Once a trace is kept for this duration, it's then released to the next consumer and removed from the internal storage. Spans from a trace that has been released will be kept for the entire duration again. In `emit_strategy: service` mode, this instead applies to subtraces.

The `num_workers` (default=1) property controls how many concurrent workers the processor will use to process traces. If you are looking to optimize this value
then using GOMAXPROCS could be considered as a starting point.

The `emit_strategy` (default=`"trace"`) property controls the span-emission granularity:

| Value | Behaviour |
| - | - |
| `"trace"` (default) | Buffer all spans for a trace and release them together after `wait_duration`. |
| `"service"` | Buffer the spans a trace passed through each service and release them separately after `wait_duration`, one batch per call to a service. |

## Subtrace Emit

When `emit_strategy: service` is set, the processor groups spans at service granularity rather than trace granularity: the spans a trace passed through one service are buffered together and released after `wait_duration`, independently of the rest of the trace.

This is useful for reducing latency when a downstream consumer performs processing for all spans from a given service within a trace and it isn't necessary to wait for the full distributed trace to arrive. The `wait_duration` option can sometimes be lowered when using this mode, as spans from a single service in a trace can arrive before the full trace completes, particularly in traces that contain asynchronous operations.

Spans are buffered under `(trace, service)`, where the service is taken from `service.namespace`, `service.name` and `service.instance.id` on the span's resource, falling back to a hash of every resource attribute when `service.name` is absent. Other resource attributes are not considered for determining service identity.

### Separating repeat calls

A trace can pass through the same service more than once, for example `A -> B -> C -> B`. Each of those is a separate call and is released as its own batch, so a batch never stands for two calls a consumer could otherwise have told apart.

Each call waits out `wait_duration` from its own first span. A trace that comes back to a service some time after it first passed through therefore gets a full window for the later visit, instead of inheriting the deadline of the earlier one and being cut off partway.

Which spans belong to which call is worked out when the buffer is released, not as spans arrive, because until then the picture is incomplete: a span may turn up before its parent does. A span begins a call when

1. its parent span ID is empty, or
2. the `IS_REMOTE` flag is set in `Span.flags`, which marks a service-entry span even when the parent carries the same service identity, or
3. its parent is buffered under a different service, or is a span the trace has already released.

Everything else descends from one of those spans and travels with it.

### Spans with no parent

A span whose parent is nowhere in the trace, because it is in a different Collector instance or was never sent, gets best-effort handling: there is nothing to tell one such span apart from another in the same service, so they leave together in a single batch rather than one batch per span. That batch may therefore stand for more than one call, which is the price of the information not being there. The alternative loses the grouping altogether, and the point of releasing them is to clear the buffer without dropping data.

Malformed input in which a span is its own ancestor leaves a ring of spans that nothing heads. Those are released along with the rest of their service, so no shape of input leaves spans buffered indefinitely.

### Limits

`num_traces` bounds how many `(trace, service)` groups are buffered at once, whatever number of separate calls a group holds. When that is exceeded the oldest group is released early rather than dropped: eviction is there to bound how much is held, and passing the spans on achieves that without losing them. Such evictions are counted in `otelcol_processor_groupbytrace_traces_evicted`, so a non-zero count still means `wait_duration` or `num_traces` wants adjusting. Note this differs from `emit_strategy: trace`, where an evicted trace's spans are discarded. Anything still buffered at shutdown is likewise flushed to the next consumer.

## Metrics

The following metrics are recorded by this processor:

* `otelcol_processor_groupbytrace_conf_num_traces` represents the maximum number of traces that can be kept by the internal storage. This value comes from the processor's configuration and will never change over the lifecycle of the processor.
* `otelcol_processor_groupbytrace_event_latency_bucket`, with the following `event` tag values:
  * `onTraceReceived` represents the number of traces' parts the processor has received from the previous components
  * `onTraceExpired` represents the number of traces that finished waiting in memory for spans to arrive
  * `onTraceReleased` represents the number of traces that have been marked as released to the next component
  * `onTraceRemoved` represents the number of traces that have been marked for removal from the internal storage
  * `subtrace_tick` is the only event `emit_strategy: service` adds. A worker keeps one timer for all the subtraces it holds, so this covers all of the work the three events above do in `emit_strategy: trace`: it is recorded once per wake-up, whatever number of subtraces came due, rather than once per subtrace
* `otelcol_processor_groupbytrace_num_events_in_queue` representing the state of the internal queue. Ideally, this number would be close to zero, but might have temporary spikes if the storage is slow.
* `otelcol_processor_groupbytrace_num_traces_in_memory` representing the state of the internal trace storage, waiting for spans to arrive. It's common to have items in memory all the time if the processor has a continuous flow of data. The longer the `wait_duration`, the higher the amount of traces in memory should be, given enough traffic.
* `otelcol_processor_groupbytrace_spans_released` and `otelcol_processor_groupbytrace_traces_released` represent the number of spans and traces effectively released to the next component.
* `otelcol_processor_groupbytrace_traces_evicted` represents the number of traces that have been evicted from the internal storage due to capacity problems. Ideally, this should be zero, or very close to zero at all times. If you keep getting items evicted, increase the `num_traces`.
* `otelcol_processor_groupbytrace_incomplete_releases` represents the traces that have been marked as expired, but had been previously been removed. This might be the case when a span from a trace has been received in a batch while the trace existed in the in-memory storage, but has since been released/removed before the span could be added to the trace. This should always be very close to 0, and a high value might indicate a software bug.

When `emit_strategy: service` is configured, the same metrics are emitted for subtraces: `otelcol_processor_groupbytrace_traces_released` counts released calls, so a service entered twice in one trace counts twice, `otelcol_processor_groupbytrace_spans_released` counts their spans, `otelcol_processor_groupbytrace_traces_evicted` counts `(trace, service)` groups evicted and released early, and `otelcol_processor_groupbytrace_incomplete_releases` counts subtraces that came due for release after being deleted from storage. As in `emit_strategy: trace`, that should stay at zero, and a non-zero count indicates a software bug rather than an eviction or a late span.

A healthy system would have the same value for the metric `otelcol_processor_groupbytrace_spans_released` and for three events under `otelcol_processor_groupbytrace_event_latency_bucket`: `onTraceExpired`, `onTraceRemoved` and `onTraceReleased`.

The metric `otelcol_processor_groupbytrace_event_latency_bucket` is a bucket and shows how long each event took to be processed in milliseconds. In most cases, it should take less than 5ms for an event to be processed, but it might be the case where an event could take 10ms. Higher latencies are possible, but it should never really reach the last item, representing 1s. Events taking more than 1s are killed automatically, and if you have multiple items in this bucket, it might indicate a bug in the software.

Most metrics are updated when the events occur, except for the following ones, which are updated periodically:

* `otelcol_processor_groupbytrace_num_events_in_queue`
* `otelcol_processor_groupbytrace_num_traces_in_memory`

## Warnings

* [Statefulness](https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/standard-warnings.md#statefulness): The groupbytrace processor's works best when all spans for a trace are sent to the same collector instance.

## Configuration

### Example Configuration

```yaml theme={null}
groupbytrace/custom:
  wait_duration: 10s
  num_traces: 1000
```

***

*Last generated: 2026-10-05*