> ## Documentation Index
> Fetch the complete documentation index at: https://otel.fyi/llms.txt
> Use this file to discover all available pages before exploring further.

# Drain

> OpenTelemetry processor for Drain

# Drain Processor

![Status](https://img.shields.io/badge/status-alpha-red)

**Available in:** `contrib`, `k8s`

**Maintainers:** [@MikeGoldsmith](https://github.com/MikeGoldsmith), [@atoulme](https://github.com/atoulme), [@martinjt](https://github.com/martinjt)

**Source:** [opentelemetry-collector-contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/drainprocessor)

## Supported Telemetry

![Logs](https://img.shields.io/badge/logs-alpha-blue)

## Overview

This processor **annotates**; it does not filter. Use the [filter processor](../filterprocessor/README.md) downstream to act on the `log.record.template` attribute — for example, to drop entire classes of noisy logs by pattern.

## How it works

Drain builds a parse tree from the token structure of log lines. Lines with similar structure are grouped into a **cluster**, and a **template** is derived by replacing variable tokens with `<*>` wildcards. As more logs arrive the templates become more accurate and stable.

Use the template **string** for filtering rules; it converges to the same value across instances given the same configuration and log patterns (see [Deployment considerations](#deployment-considerations)).

## Configuration

```yaml theme={null}
processors:
  drain:
    # Drain parse tree parameters
    tree_depth: 4              # default: 4 (minimum: 3; `depth` in the Drain paper)
    merge_threshold: 0.4       # default: 0.4, range [0.0, 1.0] (`st` in the Drain paper)
    max_node_children: 100     # default: 100 (`maxChild` in the Drain paper)
    max_clusters: 0            # default: 0 (unlimited, LRU eviction when > 0)
    extra_delimiters: []       # default: [] (extra token delimiters beyond whitespace)

    # Body extraction
    body_field: ""             # default: "" (use full body string)

    # Output attribute name
    template_attribute: "log.record.template"    # default

    # Parameter extraction (optional)
    masking_rules: []                                              # default: []
    parameter_key_prefix: "log.record.template.parameter"          # default
    emit_wildcards: false                                          # default: false
    wildcards_attribute: "log.record.template.wildcards"           # default

    # Seeding (optional)
    seed_templates: []
    seed_logs: []

    # Warmup suppression (optional)
    warmup_min_clusters: 0     # default: 0 (disabled; annotates from the first record)

    # Snapshot persistence (optional)
    storage: ""                # default: "" (disabled; ID of a storage extension)
    save_interval: 0s          # default: 0s (save on shutdown only; e.g. "5m" for periodic saves)
```

### Parameters

| Field | Type | Default | Description |
| - | - | - | - |
| `tree_depth` | int | `4` | Max depth of the Drain parse tree (`depth` in the Drain paper). Higher values produce more specific templates. Minimum: 3. |
| `merge_threshold` | float | `0.4` | Minimum fraction of tokens that must match an existing cluster template for a log line to be merged into it rather than forming a new cluster (`st` in the Drain paper). Range: \[0.0, 1.0]. |
| `max_node_children` | int | `100` | Maximum children per internal parse tree node (`maxChild` in the Drain paper). Bounds memory on high-cardinality token positions. |
| `max_clusters` | int | `0` | Maximum clusters tracked. When exceeded, the least-recently-used cluster is evicted. `0` means unlimited. |
| `extra_delimiters` | \[]string | `[]` | Additional token delimiters beyond whitespace (e.g. `[",", ":"]`). |
| `body_field` | string | `""` | If set, and the log body is a structured map, the value of this top-level key is used as the text to template instead of the full body. |
| `template_attribute` | string | `"log.record.template"` | Attribute key written with the derived template string. |
| `masking_rules` | \[]object | `[]` | Ordered list of `{name, pattern}` rules applied to a working copy of the log body before it is fed to the Drain tree (see [Parameter extraction](#parameter-extraction)). Templates surface named mask tokens (e.g. `<ip>`); each matched position writes a dynamic attribute at `<parameter_key_prefix>.<name>`. |
| `parameter_key_prefix` | string | `"log.record.template.parameter"` | Attribute-key prefix for extracted named parameters. Each masked position writes to `<parameter_key_prefix>.<mask name>` with the raw body value. Only consulted when `masking_rules` is non-empty. |
| `emit_wildcards` | bool | `false` | When `true`, writes a positional string slice attribute containing body tokens at each Drain `<*>` position, in template order. Independent of `masking_rules`. |
| `wildcards_attribute` | string | `"log.record.template.wildcards"` | Attribute key for the wildcards slice. Only consulted when `emit_wildcards` is `true`. |
| `seed_templates` | \[]string | `[]` | Template strings to pre-load at startup (see [Seeding](#seeding)). |
| `seed_logs` | \[]string | `[]` | Raw example log lines to train on at startup (see [Seeding](#seeding)). |
| `warmup_min_clusters` | int | `0` | Number of distinct clusters that must be observed before annotation is enabled. `0` disables warmup suppression (see [Warmup suppression](#warmup-suppression)). |
| `storage` | string | `""` | ID of a [storage extension](../../extension/storage/) to use for persisting the Drain tree across restarts (see [Snapshot persistence](#snapshot-persistence)). |
| `save_interval` | duration | `0s` | Interval between periodic snapshot saves. `0s` saves on shutdown only. Requires `storage` to be set. |

## Seeding

Seeding pre-populates the Drain tree before any live logs arrive. This is the primary mechanism for stable templates across restarts.

### `seed_templates`

Provide known template strings directly. The processor trains on each entry at startup, establishing clusters for those patterns immediately.

```yaml theme={null}
processors:
  drain:
    seed_templates:
      - "user <*> logged in from <*>"
      - "connected to <*>"
      - "heartbeat ping <*>"
```

### `seed_logs`

Provide raw example log lines. The processor trains on them at startup, letting Drain derive the templates itself. Useful when exact template strings are not known in advance.

```yaml theme={null}
processors:
  drain:
    seed_logs:
      - "user alice logged in from 10.0.0.1"
      - "user bob logged in from 192.168.1.1"
      - "connected to 10.0.0.1"
```

Empty and whitespace-only entries in both lists are silently skipped.

## Deployment considerations

### Multiple collector instances

Each collector instance builds its Drain parse tree independently in memory. Two instances processing the same log patterns will converge on identical templates because the Drain algorithm is deterministic: given the same configuration and a representative sample of log forms, the same token structure produces the same template string.

The main caveat is the **early training phase**. Before an instance has seen enough lines to abstract a wildcard (e.g. before `"user alice logged in"` and `"user bob logged in"` have both been observed), different instances may temporarily produce different templates for the same logical pattern. This is most noticeable at startup with low-volume or highly variable log streams.

**Mitigations:**

* Use `seed_templates` or `seed_logs` to pre-load known patterns at startup. With a comprehensive seed set, instances start in an already-converged state and live training only fills in the gaps.
* Use `warmup_min_clusters` to suppress annotation until the tree has stabilised, avoiding unstable templates reaching downstream processors.

### Warmup suppression

When `warmup_min_clusters` is set to a value greater than zero, the processor trains on every record from the start but does not write `log.record.template` until that many distinct clusters have been observed. Records pass through immediately — there is no buffering or added latency — they simply arrive at the next processor unannotated during the warmup window.

The warmup window is observable via `otelcol_processor_incoming_items - otelcol_processor_drain_log_records_annotated` — the difference represents records that passed through without a template attribute.

```yaml theme={null}
processors:
  drain:
    warmup_min_clusters: 20
```

Once the threshold is reached, all subsequent records are annotated normally. Records that passed through during warmup are not re-annotated.

## Metrics

The processor emits the following internal telemetry metrics:

| Metric | Type | Description |
| - | - | - |
| `otelcol_processor_drain_clusters_active` | gauge | Current number of active clusters in the Drain parse tree. Useful for tracking tree growth and stability over time. |
| `otelcol_processor_drain_log_records_annotated` | counter | Number of log records successfully annotated with a template. |
| `otelcol_processor_drain_masks_duplicates` | counter | Number of records where a mask name matched more than one position in the matched template. Incremented once per record per duplicated mask name and tagged with a `mask` attribute naming the offending rule. See [Duplicate mask names](#duplicate-mask-names). |

## Output attributes

The processor sets the following attributes on each log record:

| Attribute | Type | Example | Description |
| - | - | - | - |
| `log.record.template` | string | `"user <*> logged in from <ip>"` | The Drain-derived template string. Stable within an instance once the tree has warmed up. Use this for filtering rules. |
| `log.record.template.parameter.<name>` | string | `log.record.template.parameter.ip = "10.0.0.1"` | Optional. One attribute per mask name that matched in the template; the value is the raw body token at that position. The `log.record.template.parameter` prefix is configurable via `parameter_key_prefix`. Written only when `masking_rules` is non-empty. |
| `log.record.template.wildcards` | \[]string | `["alice", "42"]` | Optional. Positional body tokens at Drain's `<*>` positions in template order. Written only when `emit_wildcards` is `true`. |

The attribute names are configurable via `template_attribute`, `parameter_key_prefix`, and `wildcards_attribute`.

> **Semantic conventions**: `log.record.template` aligns with the proposed OTel attribute in [open-telemetry/semantic-conventions#1283](https://github.com/open-telemetry/semantic-conventions/issues/1283) and [#2064](https://github.com/open-telemetry/semantic-conventions/issues/2064). The `log.record.template.parameter.<key>` shape matches the pattern used by `http.request.header.<key>` and `db.query.parameter.<key>`. These names may be updated if a convention is formally adopted.

## Parameter extraction

Configure `masking_rules` to name variable positions and stabilise the tree on high-cardinality values. Each rule declares a regex whose matches in a working copy of the body are replaced with the literal mask token `<name>` before Drain sees the line. The original body is never mutated.

```yaml theme={null}
processors:
  drain:
    masking_rules:
      - name: ip
        pattern: '(\d{1,3}\.){3}\d{1,3}'
    emit_wildcards: true
```

Given the body `"user alice logged in from 10.0.0.1 in 42 ms"` and the rule above, Drain sees `"user alice logged in from <ip> in 42 ms"` and produces a template such as `"user <*> logged in from <ip> in <*> ms"`. The resulting attributes are:

```
log.record.template                       = "user <*> logged in from <ip> in <*> ms"
log.record.template.parameter.ip          = "10.0.0.1"
log.record.template.wildcards             = ["alice", "42"]
```

### How it works

1. The rules run in declaration order. Each rule's `pattern` is a Go regular expression (RE2 syntax). Matches are substituted with `<name>` on the output of the previous rule, so earlier rules effectively win on overlapping input text. Capture groups are permitted but ignored; the whole match is replaced.
2. Drain trains on the masked line. Mask tokens appear as literal tokens in the template alongside Drain's own `<*>` wildcards.
3. Each mask-matched position writes a dynamic attribute at `<parameter_key_prefix>.<mask name>` with the raw body value.
4. When `emit_wildcards: true`, Drain's own `<*>` positions are written as a positional string slice at `wildcards_attribute`, in template order.

### Discovery workflow

Enable `emit_wildcards: true` without any `masking_rules` to see the raw variable values in each template:

```yaml theme={null}
processors:
  drain:
    emit_wildcards: true
```

Records get `log.record.template.wildcards` populated with the raw tokens at every `<*>` position. Once you see which values look like well-known types (IPs, UUIDs, numbers), add matching `masking_rules` to promote them to named attributes.

### Duplicate mask names

When a mask rule matches multiple positions in the same template (e.g. `"connection from <ip> to <ip>"`), only one value can be written under `<parameter_key_prefix>.<name>`. First-match wins: the earliest position's value is written and later positions for the same mask name are dropped.

Each dropped position increments the `otelcol_processor_drain_masks_duplicates` counter (once per record per duplicated mask name), tagged with the offending mask name via the `mask` attribute. Watch this metric to spot rules that need splitting into distinct names, e.g. `src_ip` and `dst_ip` instead of a single `ip`.

### Constraints

* Mask names must be non-empty, must not contain `<`, `>`, or whitespace, and must not equal `*` (reserved for Drain's wildcard).
* A mask pattern that spans whitespace collapses multiple raw tokens into a single masked token. When this happens the raw body and the template can no longer be aligned position-by-position; both parameter attributes and the wildcards slice are skipped for that record. The template attribute is still written. Prefer patterns that match within a single whitespace-delimited token.
* Body tokenisation matches the parse tree's: whitespace plus any `extra_delimiters`.
* During warmup suppression no `log.record.template*` attribute is written.

### Interaction with seed data

Entries in `seed_logs` go through the same masking pass as live records, so the tree learns the same shape from seeds and live traffic. Entries in `seed_templates` are trained verbatim; the user is declaring template shape, so any mask tokens they include appear as-is.

## Example pipeline

The following pipeline annotates logs with Drain templates and then drops known noisy patterns using the filter processor:

```yaml theme={null}
processors:
  drain:
    tree_depth: 4
    merge_threshold: 0.4
    max_clusters: 500
    seed_templates:
      - "user <*> logged in from <*>"
      - "connected to <*>"
      - "heartbeat ping <*>"
    warmup_min_clusters: 20

  filter/drop_noisy:
    error_mode: ignore
    logs:
      log_record:
        - attributes["log.record.template"] == "heartbeat ping <*>"
        - attributes["log.record.template"] == "connected to <*>"

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [drain, filter/drop_noisy]
      exporters: [otlp]
```

## `body_field`

`body_field` is a convenience for pipelines where the log body is a structured map and you do not have full control over how upstream processors shape it.

If you **do** control the pipeline, the preferred approach is a `move` operator in the filelog receiver (or equivalent) to promote the message field back to a plain string body before the drain processor sees the record:

```yaml theme={null}
operators:
  - type: json_parser
  - type: move
    from: body.message
    to: body
```

If you **cannot** do that — for example, logs arrive via OTLP already structured — set `body_field` to the map key whose value should be fed to Drain:

```yaml theme={null}
processors:
  drain:
    body_field: "message"
```

Given a log body `{"level": "info", "message": "user alice logged in from 10.0.0.1"}`, only the `message` value is fed to Drain. The full body is used unchanged if the field is absent or the body is not a map.

> **Note**: `body_field` only supports a single top-level key. Full OTTL path expressions (e.g. `body["event"]["message"]`) are not supported and are noted as a future extension.

## Snapshot persistence

When `storage` is set to the ID of a [storage extension](../../extension/storage/), the processor saves and restores the Drain tree state across restarts. This eliminates the need for seeding and warmup in most deployments.

```yaml theme={null}
extensions:
  file_storage:
    directory: /var/lib/otelcol/drain

processors:
  drain:
    storage: file_storage
    save_interval: 5m
```

### Lifecycle

* **Startup**: the processor attempts to load a snapshot from storage. If a valid snapshot is found, `seed_templates` and `seed_logs` are skipped (they are already incorporated in the snapshot). If the loaded tree has enough clusters to satisfy `warmup_min_clusters`, warmup is also skipped.
* **Runtime**: if `save_interval` is set, the tree is snapshotted periodically. A hash check avoids redundant writes when the tree has not changed.
* **Shutdown**: a final snapshot is always saved.

If the snapshot is corrupt or cannot be loaded, the processor logs a warning and falls back to seeding as if no snapshot existed.

### Shared storage for scaled deployments

With a shared storage backend (Redis, database), periodic saves allow new instances joining a scaled deployment to load a tree that was trained by already-running instances. This avoids the cold-start problem where newly added collectors must rebuild the tree from scratch.

Multiple instances writing to the same storage key is safe — all instances see similar log traffic and converge on similar trees, so any snapshot is useful as a starting point. The hash check reduces redundant writes across instances.

To clear the snapshot and force a fresh start (e.g. after changing Drain parameters), delete the storage data for the processor.

## Future extensions

* **OTTL body extraction**: support full OTTL path expressions for `body_field` instead of a single top-level key name.
* **Multi-instance tree merging**: gossip-based tree merging for consistent templates across horizontally scaled deployments.

## Attributes

| Attribute Name | Description | Type | Values |
| - | - | - | - |
| `mask` | The mask name that matched more than one position in a single template. | string | |

***

*Last generated: 2026-10-05*