Spanpruning Processor
contrib
Maintainers: @portertech, @csmarchbanks
Source: opentelemetry-collector-contrib
Supported Telemetry
Overview
Overview
The Span Pruning Processor identifies duplicate or similar leaf spans within a single trace, groups them, and replaces each group with a single aggregated summary span. When leaf spans are aggregated, the processor also recursively aggregates their parent spans if all children of those parents are being aggregated. Leaf spans are spans that are not referenced as a parent by any other span in the trace. They typically represent the last actions in an execution call stack (e.g., individual database queries, HTTP calls to external services). Spans are grouped by:- Span name - spans must have the same name
- Span kind - spans must have the same kind (Internal, Server, Client, Producer, Consumer)
- Status code - spans must have the same status (OK, Error, or Unset)
- TraceState - spans must have identical TraceState values (for Consistent Probability Sampling compatibility)
- Configured attributes - spans must have matching values for attributes specified in
group_by_attributes - Parent span name - leaf spans must share the same parent span name to be grouped together
Use Cases
- Database query optimization: When an application makes many similar database queries (e.g., N+1 queries), aggregate them into a single summary span
- Batch operations: Consolidate many similar leaf operations into a single representative span
- Cost reduction: Reduce trace storage costs by eliminating redundant span data
Configuration
Configuration Options
| Field | Type | Default | Description |
|---|---|---|---|
group_by_attributes | []string | [] | Attribute patterns for grouping (supports glob patterns like db.*) |
min_spans_to_aggregate | int | 5 | Minimum group size before aggregation occurs |
max_parent_depth | int | 1 | Max depth of parent aggregation (0=none, -1=unlimited) |
aggregation_attribute_prefix | string | ”aggregation.” | Prefix for aggregation statistics attributes |
aggregation_histogram_buckets | []time.Duration | [5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s, 10s] | Upper bounds for latency histogram buckets |
enable_attribute_loss_analysis | bool | false | Enable attribute loss analysis (records attribute-loss metrics and adds summary span loss attributes) |
attribute_loss_exemplar_sample_rate | float64 | 0 (disabled) | Fraction of attribute-loss metric recordings with exemplars (0.0-1.0). Only applies when enable_attribute_loss_analysis is true. |
enable_outlier_analysis | bool | false | Enable outlier detection and correlation analysis |
outlier_analysis.method | string | ”iqr” | Statistical method: “iqr” or “mad” |
outlier_analysis.iqr_multiplier | float64 | 1.5 | IQR threshold multiplier (when method=iqr) |
outlier_analysis.mad_multiplier | float64 | 3.0 | MAD threshold multiplier (when method=mad) |
outlier_analysis.min_group_size | int | 7 | Minimum group size for outlier analysis |
outlier_analysis.correlation_min_occurrence | float64 | 0.75 | Minimum outlier occurrence fraction for correlation |
outlier_analysis.correlation_max_normal_occurrence | float64 | 0.25 | Maximum normal occurrence fraction for correlation |
outlier_analysis.max_correlated_attributes | int | 5 | Maximum correlated attributes to report |
outlier_analysis.preserve_outliers | bool | false | Keep outliers as individual spans instead of aggregating |
outlier_analysis.max_preserved_outliers | int | 2 | Max outliers to preserve per group (0=preserve all) |
outlier_analysis.preserve_only_with_correlation | bool | false | Only preserve outliers if a strong correlation is found |
outlier_analysis.min_outlier_threshold_percent | float64 | 0.1 | Minimum percentage above median required before a span is considered an outlier |
Glob Pattern Support
Thegroup_by_attributes field supports glob patterns for matching attribute keys:
| Pattern | Matches |
|---|---|
db.* | db.operation, db.name, db.statement, etc. |
http.request.* | http.request.method, http.request.header.content-type, etc. |
rpc.* | rpc.method, rpc.service, rpc.system, etc. |
db.operation | Only the exact key db.operation |
Summary Span
When spans are aggregated, the summary span includes:Properties
- Name: Original span name (e.g.,
SELECT) - TraceID: Same as original spans
- SpanID: Newly generated unique ID
- ParentSpanID: Same as original spans (common parent)
- Kind: Same as template span (inherited from slowest span)
- StartTimestamp: Earliest start time of all spans in the group
- EndTimestamp: Latest end time of all spans in the group
- Status: Same as original spans (spans are grouped by status code)
- TraceState: Inherited from the template span (preserved for Consistent Probability Sampling compatibility)
- Attributes: Inherited from the slowest span in the group
Note: The summary span’s duration (EndTimestamp - StartTimestamp) represents the total time window covered by all aggregated spans, which may exceedduration_max_ns. For example, if spans overlap or are staggered, the time range can be larger than any individual span’s duration. Useduration_max_nsto find the slowest individual operation.
What Gets Aggregated Away
When spans are aggregated into a summary span, the following data from non-template spans is lost:| Data | Behavior |
|---|---|
| Span Events | Events from the template (slowest) span are preserved |
| Span Links | Links from the template span are preserved |
| Attributes | Non-matching attribute values are lost |
| Individual Timestamps | Original start/end times replaced by the group’s time range |
| SpanIDs | Original SpanIDs are replaced by a single summary SpanID |
Aggregation Attributes
The following attributes are added to the summary span (shown with defaultaggregation_attribute_prefix: "aggregation."):
| Attribute | Type | Description |
|---|---|---|
<prefix>is_summary | bool | Always true to identify summary spans |
<prefix>span_count | int64 | Number of spans that were aggregated |
<prefix>duration_min_ns | int64 | Minimum duration in nanoseconds |
<prefix>duration_max_ns | int64 | Maximum duration in nanoseconds |
<prefix>duration_avg_ns | int64 | Average duration in nanoseconds |
<prefix>duration_total_ns | int64 | Total duration in nanoseconds |
<prefix>histogram_bucket_bounds_s | []float64 | Bucket upper bounds in seconds (excludes +Inf) |
<prefix>histogram_bucket_counts | []int64 | Cumulative count per bucket (includes +Inf bucket) |
Optional Attribute Loss Analysis
Whenenable_attribute_loss_analysis: true, the processor analyzes how attributes vary across aggregated spans and records two loss types:
- Diversity loss: Attribute exists on all spans in the group, but values differ across spans.
- Missing loss: Attribute is absent from some spans in the group.
aggregation_attribute_prefix):
| Attribute | Type | Description |
|---|---|---|
<prefix>diverse_attributes | string | Attributes with value diversity loss. Format: key(lost_values),... |
<prefix>missing_attributes | string | Attributes missing on some spans. Format: key(lost_values),... |
processor_spanpruning_leaf_attribute_diversity_lossprocessor_spanpruning_leaf_attribute_lossprocessor_spanpruning_parent_attribute_diversity_lossprocessor_spanpruning_parent_attribute_loss
attribute_loss_exemplar_sample_rate to control how often those metric points include exemplars for trace correlation.
Optional Outlier Analysis Attributes
Whenenable_outlier_analysis: true, the following additional attributes are added:
| Attribute | Type | Description |
|---|---|---|
<prefix>duration_median_ns | int64 | Median duration (more robust than average for skewed distributions) |
<prefix>outlier_correlated_attributes | string | Attributes that distinguish outliers from normal spans (format: key=value(outlier%/normal%), ...) |
Summary Span Attributes (When Preserving Outliers)
Whenoutlier_analysis.preserve_outliers: true, the summary span also includes:
| Attribute | Type | Description |
|---|---|---|
<prefix>preserved_outlier_count | int64 | Number of outlier spans preserved |
<prefix>preserved_outlier_span_ids | []string | SpanIDs of preserved outliers |
Preserved Outlier Span Attributes
Preserved outlier spans are annotated with:| Attribute | Type | Description |
|---|---|---|
<prefix>is_preserved_outlier | bool | Identifies span as a preserved outlier |
<prefix>summary_span_id | string | SpanID of the associated summary span |
summary_span_id) sits where its group was.
Histogram Buckets
Whenaggregation_histogram_buckets is configured, summary spans include latency distribution data as cumulative histogram buckets. Cumulative means each bucket count includes all spans with duration less than or equal to that bucket boundary.
Worked example with buckets [10ms, 50ms, 100ms] and span durations [5ms, 15ms, 25ms, 75ms, 150ms]:
histogram_bucket_bounds_s:[0.01, 0.05, 0.1]histogram_bucket_counts:[1, 3, 4, 5]- Bucket 0 (<=10ms): 1 span (5ms)
- Bucket 1 (<=50ms): 3 spans (5ms, 15ms, 25ms)
- Bucket 2 (<=100ms): 4 spans (5ms, 15ms, 25ms, 75ms)
- Bucket 3 (+Inf): 5 spans (all spans)
Outlier Analysis (Optional)
Whenenable_outlier_analysis: true, the processor detects duration outliers and identifies attributes that correlate with slow spans.
Detection Methods
The processor supports two statistical methods for outlier detection:| Method | Formula | Characteristics |
|---|---|---|
| IQR (default) | threshold = Q3 + (multiplier × IQR) | Standard method; sensitive to moderate outliers; uses quartiles |
| MAD | threshold = median + (multiplier × MAD × 1.4826) | More robust to extreme outliers; uses median |
- IQR: Best for typical distributions with moderate outliers. Standard choice for most use cases.
- MAD: Better when you have extreme outliers that would skew IQR calculations, or when you need more stable detection thresholds.
How It Works
IQR (Interquartile Range) Method:- Sort spans by duration
- Calculate Q1 (25th percentile) and Q3 (75th percentile)
- Calculate IQR = Q3 - Q1
- Flag spans with duration > Q3 + (iqr_multiplier × IQR) as outliers
- Sort spans by duration and find the median
- Calculate |duration - median| for each span
- MAD = median of those deviations
- Flag spans with duration > median + (mad_multiplier × MAD × 1.4826) as outliers
- Compare attribute values between outliers and normal spans
- Find attribute values that appear frequently in outliers but rarely in normal spans
- Report the strongest correlations based on the configured thresholds
Configuration Example
Example Output
- Median vs Avg: Large difference (8ms vs 45ms) indicates outliers are skewing the average
- Primary correlation: All outliers (100%) had
cache_hit=false, while 0% of normal spans did - Secondary correlation: 80% of outliers hit shard 7, but only 10% of normal spans did
- Cache misses
- Specific database shards
- Failed retries
- Timeout scenarios
When to Use
- Enable when you need to understand why some operations are slow
- Disable (default) to minimize overhead when outlier analysis isn’t needed
- Works best with groups of 10+ spans for statistical reliability
Performance Impact
- Computational overhead: Sorts durations, calculates quartiles, counts attribute occurrences
- Minimal when disabled: Zero overhead (no sorting or calculations)
- Recommended: Use
min_group_size: 7or higher to skip analysis on small groups
Preserving Outlier Spans (Optional)
Whenoutlier_analysis.preserve_outliers: true, detected outlier spans are kept as individual spans instead of being aggregated. This provides:
- Full visibility into slow operations for debugging
- Preserved context: Original attributes, events, and links remain intact
- Selective aggregation: Only prune repetitive normal spans
Configuration
Configuration Options
| Field | Type | Default | Description |
|---|---|---|---|
preserve_outliers | bool | false | Keep outliers as individual spans instead of aggregating |
max_preserved_outliers | int | 2 | Max outliers to preserve per group (0=preserve all detected) |
preserve_only_with_correlation | bool | false | Only preserve outliers if a strong attribute correlation is found |
Example Output
Before (10 similar SELECT spans, 2 are outliers):preserve_outliers: true, max_preserved_outliers: 2):
Behavior Notes
- Parent aggregation: Parents can still be aggregated if all their children are either aggregated or preserved as outliers
- Skip aggregation: If preserving outliers leaves too few normal spans (below
min_spans_to_aggregate), the entire group is left unchanged - Selection order: Outliers are preserved starting with the most extreme (longest duration) first
Pipeline Placement
This processor is designed to work best when placed after processors that ensure complete traces are available:Example
Basic Example
A trace with repeated database queries (some failing): Before Processing:min_spans_to_aggregate: 2):
Recursive Parent Aggregation Example
When spans are aggregated, the processor also checks if their parent spans can be aggregated. Parent spans are eligible for aggregation when:- All of their children are being aggregated
- They share the same name, kind, and status code with other eligible parents
- They are not root spans (must have a parent)
- At least 2 parents meet the criteria
min_spans_to_aggregate: 2, group_by_attributes: ["db.op"]):
| Span | Result | Reason |
|---|---|---|
| 3x handler (OK) with SELECT children | Aggregated | All children aggregated, same name+kind+status |
| 3x SELECT (OK) under handler | Aggregated | Same name + kind + status + attributes + parent name |
| 2x handler (Error) with SELECT children | Aggregated | All children aggregated, same name+kind+status |
| 2x SELECT (Error) under handler | Aggregated | Same name + kind + status + attributes + parent name |
| handler (OK) with INSERT child | Unchanged | Child not aggregated (only 1 INSERT) |
| INSERT (OK) | Unchanged | Below threshold (only 1 span) |
| worker (OK) | Unchanged | Child not aggregated |
| SELECT (OK) under worker | Unchanged | Different parent name than other SELECTs |
Limitations
- Requires complete traces for accurate leaf detection
- Summary span inherits attributes from the slowest span in the group
- Parent spans are only aggregated when ALL their children are aggregated
Consistent Probability Sampling (CPS) Compatibility
The processor is designed to be compatible with Consistent Probability Sampling (CPS). CPS uses TraceState to carry sampling metadata (ot=th:...;rv:...) where:
th(threshold) indicates the sampling probability thresholdrv(randomness value) provides consistent randomness for sampling decisions
th value) because:
- The
rvvalue affects sampling decisions - Vendor-specific keys may have semantic meaning
- Key ordering may be significant
Telemetry
The processor emits the following metrics to help monitor its operation:Counters
| Metric | Description |
|---|---|
otelcol_processor_spanpruning_spans_received | Total number of spans received by the processor |
otelcol_processor_spanpruning_spans_pruned | Total number of spans removed by aggregation |
otelcol_processor_spanpruning_aggregations_created | Total number of aggregation summary spans created |
otelcol_processor_spanpruning_traces_processed | Total number of traces processed |
otelcol_processor_spanpruning_outliers_detected | Total spans identified as outliers by analysis (when enable_outlier_analysis: true) |
otelcol_processor_spanpruning_outliers_preserved | Total outlier spans kept as individual spans (when preserve_outliers: true) |
otelcol_processor_spanpruning_outliers_correlations_detected | Total aggregation groups where outliers had correlated attributes |
Histograms
| Metric | Description |
|---|---|
otelcol_processor_spanpruning_aggregation_group_size | Distribution of the number of spans per aggregation group |
otelcol_processor_spanpruning_processing_duration | Time taken to process each batch of traces (in seconds) |
- Monitor the effectiveness of span pruning (compare
spans_receivedvsspans_pruned) - Track the compression ratio achieved by aggregation
- Identify processing bottlenecks via
processing_duration - Understand aggregation patterns via
aggregation_group_size
Configuration
Example Configuration
Last generated: 2026-07-06