Logdedup Processor

Available in: contrib, k8s Maintainers: @MikeGoldsmith Source: opentelemetry-collector-contrib

Supported Telemetry

Overview

How It Works

The user configures the log deduplication processor in the desired logs pipeline.
If the processor does not provide conditions, all logs are considered eligible for aggregation. If the processor does have configured conditions, all log entries where at least one of the conditions evaluates true are considered eligible for aggregation. Eligible identical logs are aggregated over the configured interval. Logs are considered identical if they have the same body, resource attributes, severity, and log attributes. Logs that do not match any condition in conditions are passed onward in the pipeline without aggregating.
After the interval, the processor emits a single log with the count of logs that were deduplicated. The emitted log will have the same body, resource attributes, severity, and log attributes as the original log. The emitted log will also have the following new attributes:
- log_count: The count of logs that were deduplicated over the interval. The name of the attribute is configurable via the log_count_attribute parameter.
- first_observed_timestamp: The timestamp of the first log that was observed during the aggregation interval.
- last_observed_timestamp: The timestamp of the last log that was observed during the aggregation interval.

Note: The ObservedTimestamp and Timestamp of the emitted log will be the time that the aggregated log was emitted and will not be the same as the ObservedTimestamp and Timestamp of the original logs.

Configuration

Field	Type	Default	Description
interval	duration	`10s`	The interval at which logs are aggregated. The counter will reset after each interval.
conditions	[]string	`[]`	A slice of OTTL expressions used to evaluate which log records are deduped. All paths in the log context are available to reference. Paths should be prefixed with their context name (e.g. `log.attributes["foo"]`, `resource.attributes["bar"]`). The un-prefixed form (e.g. `attributes["foo"]`) is deprecated; if used, the processor will log the rewritten conditions at startup so they can be migrated. All converters are available to use.
log_count_attribute	string	`log_count`	The name of the count attribute of deduplicated logs that will be added to the emitted aggregated log.
include_fields	[]string	`[]`	Fields to include in duplication matching. Fields can be from the log `body` or `attributes`. Nested fields must be `.` delimited. If a field contains a `.` it can be escaped by using a `\`. This option is mutually exclusive with `exclude_fields`. See example config.
timezone	string	`UTC`	The timezone of the `first_observed_timestamp` and `last_observed_timestamp` timestamps on the emitted aggregated log. The available locations depend on the local IANA Time Zone database. This page contains many examples, such as `America/New_York`.
exclude_fields	[]string	`[]`	Fields to exclude from duplication matching. Fields can be excluded from the log `body` or `attributes`. These fields will not be present in the emitted aggregated log. Nested fields must be `.` delimited. This option is `mutually exclusive` with `include_fields`. If a field contains a `.` it can be escaped by using a `\` see example config. Note: The entire `body` cannot be excluded. If the body is a map then fields within it can be excluded.
metadata_keys	[]string	`[]`	A list of client metadata keys (e.g. gRPC/HTTP request headers such as `x-scope-orgid`) used to partition log aggregation. Logs arriving with different values for these keys are aggregated independently and exported with a context that preserves the original metadata, allowing downstream extensions (e.g. `headers_setter`) to route them correctly. Entries are case-insensitive and duplicates are rejected. When empty (default), all logs share a single aggregation bucket.
metadata_cardinality_limit	uint32	`0`	Maximum number of distinct metadata combinations that can be tracked simultaneously. `0` means no limit (a warning is logged at startup when `metadata_keys` is set with no limit, since memory growth is unbounded). When the limit is reached, new combinations are rejected with a permanent error.

Note: The processor type has been renamed from logdedup to log_dedup. The old name is still accepted but will log a deprecation warning.

Example Config

The following config is an example configuration for the log deduplication processor. It is configured with an aggregation interval of 60 seconds, a timezone of America/Los_Angeles, and a log count attribute of dedup_count. It has no fields being excluded.

receivers:
    file_log:
        include: [./example/*.log]
processors:
    log_dedup:
        interval: 60s
        log_count_attribute: dedup_count
        timezone: 'America/Los_Angeles'
exporters:
    googlecloud:

service:
    pipelines:
        logs:
            receivers: [file_log]
            processors: [log_dedup]
            exporters: [googlecloud]

Example Config with Excluded Fields

The following config is an example configuration that excludes the following fields from being considered when searching for duplicate logs:

timestamp field from the body
host.name field from attributes
ip nested attribute inside a map attribute named src

receivers:
    file_log:
        include: [./example/*.log]
processors:
    log_dedup:
        exclude_fields:
          - body.timestamp
          - attributes.host\.name
          - attributes.src.ip
exporters:
    googlecloud:

service:
    pipelines:
        logs:
            receivers: [file_log]
            processors: [log_dedup]
            exporters: [googlecloud]

Example Config with Include Fields

This example demonstrates a configuration where deduplication is applied to telemetry based on specified fields. Only logs with the same values for the fields defined in the include_fields parameter are deduplicated:

receivers:
    file_log:
        include: [./example/*.log]
processors:
    log_dedup:
        include_fields:
          - attributes.id
          - attributes.name
        interval: 60s
        log_count_attribute: dedup_count
        timezone: 'America/Los_Angeles'
exporters:
    googlecloud:

service:
    pipelines:
        logs:
            receivers: [file_log]
            processors: [log_dedup]
            exporters: [googlecloud]

Example Config with Conditions

The following config is an example configuration that only performs the deduping process on telemetry where Attribute ID equals 1 OR where Resource Attribute service.name equals my-service:

receivers:
    file_log:
        include: [./example/*.log]
processors:
    log_dedup:
        conditions:
            - log.attributes["ID"] == 1
            - resource.attributes["service.name"] == "my-service"
        interval: 60s
        log_count_attribute: dedup_count
        timezone: 'America/Los_Angeles'
exporters:
    googlecloud:

service:
    pipelines:
        logs:
            receivers: [file_log]
            processors: [log_dedup]
            exporters: [googlecloud]

Last generated: 2026-08-24

​Logdedup Processor

​Supported Telemetry

​Overview

​How It Works

​Configuration

​Example Config

​Example Config with Excluded Fields

​Example Config with Include Fields

​Example Config with Conditions