> ## Documentation Index
> Fetch the complete documentation index at: https://otel.fyi/llms.txt
> Use this file to discover all available pages before exploring further.

# Lookup

> OpenTelemetry processor for Lookup

# Lookup Processor

![Status](https://img.shields.io/badge/status-development-orange)

**Maintainers:** [@jsvd](https://github.com/jsvd), [@dehaansa](https://github.com/dehaansa), [@VihasMakwana](https://github.com/VihasMakwana)

**Source:** [opentelemetry-collector-contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/lookupprocessor)

## Supported Telemetry

![Logs](https://img.shields.io/badge/logs-development-blue) ![Metrics](https://img.shields.io/badge/metrics-development-green) ![Traces](https://img.shields.io/badge/traces-development-orange)

## Overview

## Configuration

```yaml theme={null}
processors:
  lookup:
    source:
      type: yaml
      path: /etc/otel/mappings.yaml
    lookups:
      - key: log.attributes["user.id"]
        attributes:
          - destination: user.name
            default: "Unknown User"
```

### Full Configuration

| Field         | Description                                        | Default |
| ------------- | -------------------------------------------------- | ------- |
| `source.type` | The source type identifier (`noop`, `yaml`, `dns`) | `noop`  |
| `lookups`     | List of lookup rules (required, at least one)      | -       |

### Lookup Configuration

Each entry in `lookups` defines a lookup rule:

| Field        | Description                                                             | Default  |
| ------------ | ----------------------------------------------------------------------- | -------- |
| `key`        | OTTL value expression for extracting the lookup key (required)          | -        |
| `context`    | Default context for destination attributes: `record`, `resource`        | `record` |
| `attributes` | List of attribute mappings for writing results (required, at least one) | -        |

The `key` field supports any [OTTL value expression], including paths across contexts and converters. The path prefix depends on the signal type:

| Signal  | OTTL context    | Record-level path prefix      |
| ------- | --------------- | ----------------------------- |
| Logs    | `ottllog`       | `log.attributes["..."]`       |
| Traces  | `ottlspan`      | `span.attributes["..."]`      |
| Metrics | `ottldatapoint` | `datapoint.attributes["..."]` |

Resource attributes use `resource.attributes["..."]` for all signals. A context prefix is always required; bare `attributes["..."]` paths are rejected at configuration time.

Examples:

* `attributes["user.id"]` - record attribute (works for all signals)
* `log.attributes["user.id"]` - log record attribute (logs only)
* `span.attributes["user.id"]` - span attribute (traces only)
* `datapoint.attributes["host.id"]` - datapoint attribute (metrics only)
* `resource.attributes["service.name"]` - resource attribute (all signals)
* `Trim(attributes["raw.id"])` - apply a converter

[OTTL value expression]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/README.md#value-expressions

### Attribute Mapping

Each entry in `attributes` defines where to write a lookup result:

| Field         | Description                                                               | Default                                          |
| ------------- | ------------------------------------------------------------------------- | ------------------------------------------------ |
| `source`      | Field name in the lookup result (for map results, leave empty for scalar) | -                                                |
| `destination` | Attribute key to write the result to (required)                           | -                                                |
| `default`     | Value to use when the lookup returns no result                            | -                                                |
| `context`     | Override the key's context for this attribute                             | inherited from key's context (`record` if unset) |

## Examples

### Scalar Lookup (1:1)

When the source returns a single value per key, leave the `source` field empty:

```yaml theme={null}
processors:
  lookup:
    source:
      type: yaml
      path: /etc/otel/mappings.yaml
    lookups:
      - key: log.attributes["user.id"]
        attributes:
          - destination: user.name
            default: "Unknown User"
```

### Map Lookup (1:N)

When the source returns a map of fields per key, use the `source` field to extract individual values:

```yaml theme={null}
processors:
  lookup:
    source:
      type: yaml
      path: /etc/otel/user-details.yaml
    lookups:
      - key: log.attributes["user.id"]
        attributes:
          - source: name
            destination: user.name
            default: "Unknown"
          - source: email
            destination: user.email
          - source: role
            destination: user.role
            context: resource
```

### OTTL Converter on Key

The `key` field supports OTTL converters for transforming the lookup key before querying the source:

```yaml theme={null}
processors:
  lookup:
    source:
      type: yaml
      path: /etc/otel/mappings.yaml
    lookups:
      - key: Trim(log.attributes["raw.id"])
        attributes:
          - destination: display.name
```

### Context

* **record**: Write to the signal's record-level attributes (default). This maps to log record attributes for logs, span attributes for traces, and datapoint attributes for metrics. For metrics, lookups are evaluated for each datapoint across all metric types (Gauge, Sum, Histogram, ExponentialHistogram, Summary).
* **resource**: Write to resource attributes.

The `context` field on a key sets the default for all its destination attributes. Each attribute mapping can override this with its own `context` field.

Lookups are evaluated per record. When writing to resource attributes, later records in the same resource may overwrite values written by earlier records.

## Built-in Sources

* [noop](internal/source/noop/README.md) - No-operation source for testing
* [yaml](internal/source/yaml/README.md) - Key-value mappings from YAML files
* [dns](internal/source/dns/README.md) - DNS lookups with caching

## Caching

Sources that support external lookups (like DNS) can use the built-in LRU caching system to reduce latency and external queries. The cache uses a doubly-linked list with a hash map for O(1) lookups, insertions, and evictions.

### Cache Configuration

| Field                | Description                                                          | Default             |
| -------------------- | -------------------------------------------------------------------- | ------------------- |
| `cache.enabled`      | Enable caching                                                       | varies by source    |
| `cache.size`         | Maximum number of entries (LRU eviction, must be `> 0` when enabled) | varies by source    |
| `cache.ttl`          | Time-to-live for successful lookups                                  | `0` (no expiration) |
| `cache.negative_ttl` | TTL for "not found" results                                          | `0` (disabled)      |

### Cache Performance

Run with `go test -bench=BenchmarkCache -run=^$ ./lookupsource/`

Single-threaded (Apple M4 Pro):

| Operation             | ns/op | B/op | allocs/op |
| --------------------- | ----- | ---- | --------- |
| Get (hit)             | 36    | 0    | 0         |
| Get (negative hit)    | 36    | 0    | 0         |
| Get (miss)            | 5     | 0    | 0         |
| Set (new entry)       | 281   | 151  | 3         |
| Set (update existing) | 42    | 0    | 0         |
| Set (with eviction)   | 152   | 152  | 4         |

Concurrent (12 goroutines):

| Operation                   | ns/op | B/op | allocs/op |
| --------------------------- | ----- | ---- | --------- |
| Get (parallel)              | 131   | 13   | 1         |
| Mixed read/write (parallel) | 145   | 25   | 1         |

### Using Cache in Custom Sources

Custom sources can use the cache by wrapping their lookup function:

```go theme={null}
func createSource(ctx context.Context, settings lookupsource.CreateSettings, cfg lookupsource.SourceConfig) (lookupsource.Source, error) {
    myCfg := cfg.(*Config)

    // Create base lookup function
    lookupFn := func(ctx context.Context, key string) (any, bool, error) {
        // Perform actual lookup
        return result, true, nil
    }

    // Wrap with cache if enabled
    if myCfg.Cache.Enabled {
        cache := lookupsource.NewCache(myCfg.Cache)
        lookupFn = lookupsource.WrapWithCache(cache, lookupFn)
    }

    return lookupsource.NewSource(lookupFn, ...), nil
}
```

## Custom Sources

Custom lookup sources can be added using `WithSources`:

```go theme={null}
import (
    "github.com/open-telemetry/opentelemetry-collector-contrib/processor/lookupprocessor"
    "github.com/example/httplookup"
)

factories.Processors[lookupprocessor.Type] = lookupprocessor.NewFactoryWithOptions(
    lookupprocessor.WithSources(httplookup.NewFactory()),
)
```

### Source contract

* **Concurrency**: `Lookup` is called concurrently from multiple goroutines. Implementations must be safe for concurrent use.
* **Keys are strings**: The OTTL expression result is converted to a string before calling `Lookup`.
* **Return values**: For scalar (1:1) lookups, return any single value. For map (1:N) lookups, return `map[string]any`. Values are written to attributes via [`pcommon.Value.FromRaw`](https://pkg.go.dev/go.opentelemetry.io/collector/pdata/pcommon#Value.FromRaw). Unsupported types are stringified via `fmt.Sprintf`.
* **Errors are non-fatal**: When `Lookup` returns an error the processor logs it at Debug level and skips the lookup. It does not fail the batch.
* **Lifecycle**: `Start` is called once before any `Lookup`; `Shutdown` is called once after all processing stops. Both are optional (pass `nil` to `NewSource`).
* **Config tags**: Source config structs must use `mapstructure` struct tags. The processor decodes source configuration from a raw map using mapstructure.

### Implementing a Source

```go theme={null}
package mysource

import (
    "context"
    "errors"
    "time"

    "github.com/open-telemetry/opentelemetry-collector-contrib/processor/lookupprocessor/lookupsource"
)

type Config struct {
    Endpoint string        `mapstructure:"endpoint"`
    Timeout  time.Duration `mapstructure:"timeout"`
}

func (c *Config) Validate() error {
    if c.Endpoint == "" {
        return errors.New("endpoint is required")
    }
    return nil
}

func NewFactory() lookupsource.SourceFactory {
    return lookupsource.NewSourceFactory(
        "mysource",
        func() lookupsource.SourceConfig {
            return &Config{Timeout: 5 * time.Second}
        },
        createSource,
    )
}

func createSource(
    ctx context.Context,
    settings lookupsource.CreateSettings,
    cfg lookupsource.SourceConfig,
) (lookupsource.Source, error) {
    c := cfg.(*Config)

    return lookupsource.NewSource(
        func(ctx context.Context, key string) (any, bool, error) {
            // Perform lookup - return (value, found, error)
            return "result", true, nil
        },
        func() string { return "mysource" },
        nil, // start function (optional)
        nil, // shutdown function (optional)
    ), nil
}
```

## Benchmarks

Run benchmarks with:

```bash theme={null}
make benchmark
```

### Processor Performance

Measures the full processing pipeline including OTTL key evaluation, source lookup, value conversion, and attribute writes. Uses noop source to isolate processor overhead from source implementation (Apple M4 Pro):

| Scenario            | ns/op   | B/op    | allocs/op |
| ------------------- | ------- | ------- | --------- |
| 1 log, 1 lookup     | 396     | 728     | 22        |
| 10 logs, 1 lookup   | 1,829   | 3,538   | 94        |
| 100 logs, 1 lookup  | 17,577  | 31,732  | 814       |
| 100 logs, 3 lookups | 34,993  | 63,752  | 1,614     |
| 1000 logs, 1 lookup | 183,558 | 312,844 | 8,014     |

### YAML Source Performance

Measures only the source lookup operation (map access), isolated from processor overhead:

| Map Size       | ns/op | allocs/op |
| -------------- | ----- | --------- |
| 10 entries     | 1,462 | 0         |
| 100 entries    | 1,415 | 0         |
| 1,000 entries  | 1,388 | 0         |
| 10,000 entries | 1,368 | 0         |

***

*Last generated: 2026-07-06*