Lookup Processor

Maintainers: @jsvd, @dehaansa, @VihasMakwana Source: opentelemetry-collector-contrib

Supported Telemetry

Overview

Configuration

processors:
  lookup:
    source:
      type: yaml
      path: /etc/otel/mappings.yaml
    lookups:
      - key: log.attributes["user.id"]
        attributes:
          - destination: user.name
            default: "Unknown User"

Full Configuration

Field	Description	Default
`source.type`	The source type identifier (`noop`, `yaml`, `dns`)	`noop`
`lookups`	List of lookup rules (required, at least one)	-

Lookup Configuration

Each entry in lookups defines a lookup rule:

Field	Description	Default
`key`	OTTL value expression for extracting the lookup key (required)	-
`context`	Default context for destination attributes: `record`, `resource`	`record`
`attributes`	List of attribute mappings for writing results (required, at least one)	-

The key field supports any OTTL value expression, including paths across contexts and converters:

log.attributes["user.id"] - record attribute
resource.attributes["service.name"] - resource attribute
Trim(log.attributes["raw.id"]) - apply a converter

Attribute Mapping

Each entry in attributes defines where to write a lookup result:

Field	Description	Default
`source`	Field name in the lookup result (for map results, leave empty for scalar)	-
`destination`	Attribute key to write the result to (required)	-
`default`	Value to use when the lookup returns no result	-
`context`	Override the key’s context for this attribute	inherited from key’s context (`record` if unset)

Examples

Scalar Lookup (1:1)

When the source returns a single value per key, leave the source field empty:

processors:
  lookup:
    source:
      type: yaml
      path: /etc/otel/mappings.yaml
    lookups:
      - key: log.attributes["user.id"]
        attributes:
          - destination: user.name
            default: "Unknown User"

Map Lookup (1:N)

When the source returns a map of fields per key, use the source field to extract individual values:

processors:
  lookup:
    source:
      type: yaml
      path: /etc/otel/user-details.yaml
    lookups:
      - key: log.attributes["user.id"]
        attributes:
          - source: name
            destination: user.name
            default: "Unknown"
          - source: email
            destination: user.email
          - source: role
            destination: user.role
            context: resource

OTTL Converter on Key

The key field supports OTTL converters for transforming the lookup key before querying the source:

processors:
  lookup:
    source:
      type: yaml
      path: /etc/otel/mappings.yaml
    lookups:
      - key: Trim(log.attributes["raw.id"])
        attributes:
          - destination: display.name

Context

record: Read from and write to record-level attributes (log records, spans, metric data points) (default)
resource: Read from and write to resource attributes

The context field on a key sets the default for all its destination attributes. Each attribute mapping can override this with its own context field. Lookups are evaluated per record. When writing to resource attributes, later records in the same resource may overwrite values written by earlier records.

Built-in Sources

noop - No-operation source for testing
yaml - Key-value mappings from YAML files
dns - DNS lookups with caching

Caching

Sources that support external lookups (like DNS) can use the built-in LRU caching system to reduce latency and external queries. The cache uses a doubly-linked list with a hash map for O(1) lookups, insertions, and evictions.

Cache Configuration

Field	Description	Default
`cache.enabled`	Enable caching	varies by source
`cache.size`	Maximum number of entries (LRU eviction, must be `> 0` when enabled)	varies by source
`cache.ttl`	Time-to-live for successful lookups	`0` (no expiration)
`cache.negative_ttl`	TTL for “not found” results	`0` (disabled)

Cache Performance

Run with go test -bench=BenchmarkCache -run=^$ ./lookupsource/ Single-threaded (Apple M4 Pro):

Operation	ns/op	B/op	allocs/op
Get (hit)	36	0	0
Get (negative hit)	36	0	0
Get (miss)	5	0	0
Set (new entry)	281	151	3
Set (update existing)	42	0	0
Set (with eviction)	152	152	4

Concurrent (12 goroutines):

Operation	ns/op	B/op	allocs/op
Get (parallel)	131	13	1
Mixed read/write (parallel)	145	25	1

Using Cache in Custom Sources

Custom sources can use the cache by wrapping their lookup function:

func createSource(ctx context.Context, settings lookupsource.CreateSettings, cfg lookupsource.SourceConfig) (lookupsource.Source, error) {
    myCfg := cfg.(*Config)

    // Create base lookup function
    lookupFn := func(ctx context.Context, key string) (any, bool, error) {
        // Perform actual lookup
        return result, true, nil
    }

    // Wrap with cache if enabled
    if myCfg.Cache.Enabled {
        cache := lookupsource.NewCache(myCfg.Cache)
        lookupFn = lookupsource.WrapWithCache(cache, lookupFn)
    }

    return lookupsource.NewSource(lookupFn, ...), nil
}

Custom Sources

Custom lookup sources can be added using WithSources:

import (
    "github.com/open-telemetry/opentelemetry-collector-contrib/processor/lookupprocessor"
    "github.com/example/httplookup"
)

factories.Processors[lookupprocessor.Type] = lookupprocessor.NewFactoryWithOptions(
    lookupprocessor.WithSources(httplookup.NewFactory()),
)

Source contract

Concurrency: Lookup is called concurrently from multiple goroutines. Implementations must be safe for concurrent use.
Keys are strings: The OTTL expression result is converted to a string before calling Lookup.
Return values: For scalar (1:1) lookups, return any single value. For map (1:N) lookups, return map[string]any. Values are written to attributes via pcommon.Value.FromRaw. Unsupported types are stringified via fmt.Sprintf.
Errors are non-fatal: When Lookup returns an error the processor logs it at Debug level and skips the lookup. It does not fail the batch.
Lifecycle: Start is called once before any Lookup; Shutdown is called once after all processing stops. Both are optional (pass nil to NewSource).
Config tags: Source config structs must use mapstructure struct tags. The processor decodes source configuration from a raw map using mapstructure.

Implementing a Source

package mysource

import (
    "context"
    "errors"
    "time"

    "github.com/open-telemetry/opentelemetry-collector-contrib/processor/lookupprocessor/lookupsource"
)

type Config struct {
    Endpoint string        `mapstructure:"endpoint"`
    Timeout  time.Duration `mapstructure:"timeout"`
}

func (c *Config) Validate() error {
    if c.Endpoint == "" {
        return errors.New("endpoint is required")
    }
    return nil
}

func NewFactory() lookupsource.SourceFactory {
    return lookupsource.NewSourceFactory(
        "mysource",
        func() lookupsource.SourceConfig {
            return &Config{Timeout: 5 * time.Second}
        },
        createSource,
    )
}

func createSource(
    ctx context.Context,
    settings lookupsource.CreateSettings,
    cfg lookupsource.SourceConfig,
) (lookupsource.Source, error) {
    c := cfg.(*Config)

    return lookupsource.NewSource(
        func(ctx context.Context, key string) (any, bool, error) {
            // Perform lookup - return (value, found, error)
            return "result", true, nil
        },
        func() string { return "mysource" },
        nil, // start function (optional)
        nil, // shutdown function (optional)
    ), nil
}

Benchmarks

Run benchmarks with:

make benchmark

Processor Performance

Measures the full processing pipeline including OTTL key evaluation, source lookup, value conversion, and attribute writes. Uses noop source to isolate processor overhead from source implementation (Apple M4 Pro):

Scenario	ns/op	B/op	allocs/op
1 log, 1 lookup	396	728	22
10 logs, 1 lookup	1,829	3,538	94
100 logs, 1 lookup	17,577	31,732	814
100 logs, 3 lookups	34,993	63,752	1,614
1000 logs, 1 lookup	183,558	312,844	8,014

YAML Source Performance

Measures only the source lookup operation (map access), isolated from processor overhead:

Map Size	ns/op	allocs/op
10 entries	1,462	0
100 entries	1,415	0
1,000 entries	1,388	0
10,000 entries	1,368	0

Last generated: 2026-04-13

Collector components

​Lookup Processor

​Supported Telemetry

​Overview

​Configuration

​Full Configuration

​Lookup Configuration

​Attribute Mapping

​Examples

​Scalar Lookup (1:1)

​Map Lookup (1:N)

​OTTL Converter on Key

​Context

​Built-in Sources

​Caching

​Cache Configuration

​Cache Performance

​Using Cache in Custom Sources

​Custom Sources

​Source contract

​Implementing a Source

​Benchmarks

​Processor Performance

​YAML Source Performance