> ## Documentation Index
> Fetch the complete documentation index at: https://otel.fyi/llms.txt
> Use this file to discover all available pages before exploring further.
# Logdedup
> OpenTelemetry processor for Logdedup
# Logdedup Processor
**Available in:** `contrib`, `k8s`
**Maintainers:** [@MikeGoldsmith](https://github.com/MikeGoldsmith)
**Source:** [opentelemetry-collector-contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/logdedupprocessor)
## Supported Telemetry
## Overview
## How It Works
1. The user configures the log deduplication processor in the desired logs pipeline.
2. If the processor does not provide `conditions`, all logs are considered eligible for aggregation. If the processor does have configured `conditions`, all log entries where at least one of the `conditions` evaluates `true` are considered eligible for aggregation. Eligible identical logs are aggregated over the configured `interval`. Logs are considered identical if they have the same body, resource attributes, severity, and log attributes. Logs that do not match any condition in `conditions` are passed onward in the pipeline without aggregating.
3. After the interval, the processor emits a single log with the count of logs that were deduplicated. The emitted log will have the same body, resource attributes, severity, and log attributes as the original log. The emitted log will also have the following new attributes:
* `log_count`: The count of logs that were deduplicated over the interval. The name of the attribute is configurable via the `log_count_attribute` parameter.
* `first_observed_timestamp`: The timestamp of the first log that was observed during the aggregation interval.
* `last_observed_timestamp`: The timestamp of the last log that was observed during the aggregation interval.
**Note**: The `ObservedTimestamp` and `Timestamp` of the emitted log will be the time that the aggregated log was emitted and will not be the same as the `ObservedTimestamp` and `Timestamp` of the original logs.
## Configuration
| Field | Type | Default | Description |
| ---------------------------- | --------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| interval | duration | `10s` | The interval at which logs are aggregated. The counter will reset after each interval. |
| conditions | \[]string | `[]` | A slice of [OTTL] expressions used to evaluate which log records are deduped. All paths in the [log context] are available to reference. Paths should be prefixed with their context name (e.g. `log.attributes["foo"]`, `resource.attributes["bar"]`). The un-prefixed form (e.g. `attributes["foo"]`) is deprecated; if used, the processor will log the rewritten conditions at startup so they can be migrated. All [converters] are available to use. |
| log\_count\_attribute | string | `log_count` | The name of the count attribute of deduplicated logs that will be added to the emitted aggregated log. |
| include\_fields | \[]string | `[]` | Fields to include in duplication matching. Fields can be from the log `body` or `attributes`. Nested fields must be `.` delimited. If a field contains a `.` it can be escaped by using a `\`. This option is **mutually exclusive** with `exclude_fields`. See [example config](#example-config-with-deduplication-key). |
| timezone | string | `UTC` | The timezone of the `first_observed_timestamp` and `last_observed_timestamp` timestamps on the emitted aggregated log. The available locations depend on the local IANA Time Zone database. [This page](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) contains many examples, such as `America/New_York`. |
| exclude\_fields | \[]string | `[]` | Fields to exclude from duplication matching. Fields can be excluded from the log `body` or `attributes`. These fields will not be present in the emitted aggregated log. Nested fields must be `.` delimited. This option is `mutually exclusive` with `include_fields`. If a field contains a `.` it can be escaped by using a `\` see [example config](#example-config-with-excluded-fields).
**Note**: The entire `body` cannot be excluded. If the body is a map then fields within it can be excluded. |
| metadata\_keys | \[]string | `[]` | A list of client metadata keys (e.g. gRPC/HTTP request headers such as `x-scope-orgid`) used to partition log aggregation. Logs arriving with different values for these keys are aggregated independently and exported with a context that preserves the original metadata, allowing downstream extensions (e.g. `headers_setter`) to route them correctly. Entries are case-insensitive and duplicates are rejected. When empty (default), all logs share a single aggregation bucket. |
| metadata\_cardinality\_limit | uint32 | `0` | Maximum number of distinct metadata combinations that can be tracked simultaneously. `0` means no limit (a warning is logged at startup when `metadata_keys` is set with no limit, since memory growth is unbounded). When the limit is reached, new combinations are rejected with a permanent error. |
[OTTL]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.109.0/pkg/ottl#readme
[converters]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.109.0/pkg/ottl/ottlfuncs/README.md#converters
[log context]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.109.0/pkg/ottl/contexts/ottllog/README.md
> **Note:** The processor type has been renamed from `logdedup` to `log_dedup`. The old name is still accepted but will log a deprecation warning.
### Example Config
The following config is an example configuration for the log deduplication processor. It is configured with an aggregation interval of `60 seconds`, a timezone of `America/Los_Angeles`, and a log count attribute of `dedup_count`. It has no fields being excluded.
```yaml theme={null}
receivers:
file_log:
include: [./example/*.log]
processors:
log_dedup:
interval: 60s
log_count_attribute: dedup_count
timezone: 'America/Los_Angeles'
exporters:
googlecloud:
service:
pipelines:
logs:
receivers: [file_log]
processors: [log_dedup]
exporters: [googlecloud]
```
### Example Config with Excluded Fields
The following config is an example configuration that excludes the following fields from being considered when searching for duplicate logs:
* `timestamp` field from the body
* `host.name` field from attributes
* `ip` nested attribute inside a map attribute named `src`
```yaml theme={null}
receivers:
file_log:
include: [./example/*.log]
processors:
log_dedup:
exclude_fields:
- body.timestamp
- attributes.host\.name
- attributes.src.ip
exporters:
googlecloud:
service:
pipelines:
logs:
receivers: [file_log]
processors: [log_dedup]
exporters: [googlecloud]
```
### Example Config with Include Fields
This example demonstrates a configuration where deduplication is applied to telemetry based on specified fields. Only logs with the same values for the fields defined in the `include_fields` parameter are deduplicated:
```yaml theme={null}
receivers:
file_log:
include: [./example/*.log]
processors:
log_dedup:
include_fields:
- attributes.id
- attributes.name
interval: 60s
log_count_attribute: dedup_count
timezone: 'America/Los_Angeles'
exporters:
googlecloud:
service:
pipelines:
logs:
receivers: [file_log]
processors: [log_dedup]
exporters: [googlecloud]
```
### Example Config with Conditions
The following config is an example configuration that only performs the deduping process on telemetry where Attribute `ID` equals `1` OR where Resource Attribute `service.name` equals `my-service`:
```yaml theme={null}
receivers:
file_log:
include: [./example/*.log]
processors:
log_dedup:
conditions:
- log.attributes["ID"] == 1
- resource.attributes["service.name"] == "my-service"
interval: 60s
log_count_attribute: dedup_count
timezone: 'America/Los_Angeles'
exporters:
googlecloud:
service:
pipelines:
logs:
receivers: [file_log]
processors: [log_dedup]
exporters: [googlecloud]
```
***
*Last generated: 2026-07-06*