Skip to main content

K8sattributes Processor

Status Available in: contrib, k8s Maintainers: @dmitryax, @TylerHelmuth, @ChrsMark, @odubajDT Source: opentelemetry-collector-contrib

Supported Telemetry

Logs Metrics Traces

Overview

The processor automatically discovers k8s resources (pods), extracts metadata from them and adds the extracted metadata to the relevant spans, metrics and logs as resource attributes. The processor uses the kubernetes API to discover all pods running in a cluster, keeps a record of their IP addresses, pod UIDs and interesting metadata. The rules for associating the data passing through the processor (spans, metrics and logs) with specific Pod Metadata are configured via “pod_association” key. It represents a list of associations that are executed in the specified order until the first one is able to do the match.

Configuration

The processor stores the list of running pods and the associated metadata. When it sees a datapoint (log, trace or metric), it will try to associate the datapoint to the pod from where the datapoint originated, so we can add the relevant pod metadata to the datapoint. By default, it associates the incoming connection IP to the Pod IP. But for cases where this approach doesn’t work (sending through a proxy, etc.), a custom association rule can be specified. Each association is specified as a list of sources of associations. The maximum number of sources within an association is 4. A source is a rule that matches metadata from the datapoint to pod metadata. In order to get an association applied, all the sources specified need to match. Each sources rule is specified as a pair of from (representing the rule type) and name (representing the attribute name if from is set to resource_attribute). The following rule types are available:
  • connection: Takes the IP attribute from connection context (if available). In this case the processor must appear before any batching or tail sampling, which remove this information.
  • resource_attribute: Allows specifying the attribute name to lookup in the list of attributes of the received Resource. Semantic convention should be used for naming.
Example for a pod association configuration:
If Pod association rules are not configured, resources are associated with metadata only by connection’s IP Address. Which metadata to collect is determined by metadata configuration that defines list of resource attributes to be added. Items in the list called exactly the same as the resource attributes that will be added. The following attributes are added by default:
  • k8s.namespace.name
  • k8s.pod.name
  • k8s.pod.uid
  • k8s.pod.start_time
  • k8s.deployment.name (derived from the ReplicaSet name by default. Set the deprecated deployment_name_from_replicaset option to false to use the ReplicaSet informer for deployment name lookup.)
  • k8s.node.name
These attributes are also available for the use within association rules by default. The metadata section can also be extended with additional attributes which, if present in the metadata section, are then also available for the use within association rules. Available attributes are:
  • k8s.namespace.name
  • k8s.pod.name
  • k8s.pod.hostname
  • k8s.pod.ip
  • k8s.pod.start_time
  • k8s.pod.uid
  • k8s.replicaset.uid
  • k8s.replicaset.name
  • k8s.deployment.uid
  • k8s.deployment.name
  • k8s.daemonset.uid
  • k8s.daemonset.name
  • k8s.statefulset.uid
  • k8s.statefulset.name
  • k8s.cronjob.uid
  • k8s.cronjob.name (by default uses a Job-name heuristic when only the name is needed; the Job informer is used when k8s.cronjob.uid is enabled and/or labels or annotations are extracted with from: job)
  • k8s.job.uid
  • k8s.job.name
  • k8s.node.name
  • k8s.node.uid
  • k8s.cluster.uid
  • service.namespace
  • service.name
  • service.version(cannot be used for source rules in the pod_association when it’s calculated based on container’s image tag/digest)
  • service.instance.id(cannot be used for source rules in the pod_association)
  • Any tags extracted from the pod labels and annotations, as described in extracting attributes from pod labels and annotations
Not all the attributes are guaranteed to be added. Only attribute names from metadata should be used for pod_association’s resource_attribute, because empty or non-existing values will be ignored. Additional container level attributes can be extracted. If a pod contains more than one container, either the container.id, or the k8s.container.name attribute must be provided in the incoming resource attributes to correctly associate the matching container to the resource:
  1. If the container.id resource attribute is provided, the following additional attributes will be available:
    • k8s.container.name
    • container.image.name
    • container.image.tag
    • container.image.repo_digests (if k8s CRI populates repository digest field)
    • service.version
    • service.instance.id
  2. If the k8s.container.name resource attribute is provided, the following additional attributes will be available:
    • container.id (if the k8s.container.restart_count resource attribute is not provided, it’s not guaranteed to get the right container ID.)
    • container.image.name
    • container.image.tag
    • container.image.repo_digests (if k8s CRI populates repository digest field)
    • service.version
    • service.instance.id
  3. If the k8s.container.restart_count resource attribute is provided, it can be used to associate with a particular container instance. If it’s not set, the latest container instance will be used:
    • container.id (not added by default, has to be specified in metadata)
Please note, however, that only container.id attribute can be used for source rules in the pod_association. To use container.id in pod association, at least one container attribute must be included in the metadata extraction configuration (e.g., container.id, container.image.name, etc.). Example for extracting container level attributes:
The previous configuration attaches the attributes listed in the metadata section to all resources received by a matching pod with the k8s.container.name attribute being present. For example, when the following trace
is sent to the collector by the following pod,
the processor associates the received trace to the pod, based on the connection IP, and add those attributes to the resulting span:
By default, the processor will be ready as soon as it starts, even if no metadata has been fetched yet. If data is sent to this processor before the metadata is synced, there will be no metadata to enrich the data with. To wait for the metadata to be synced before the processor is ready, set the wait_for_metadata option to true. Then the processor will not be ready until the metadata is fully synced. As a result, the start-up of the Collector will be blocked. If the metadata cannot be synced, the Collector will ultimately fail to start. If a timeout is reached, the processor will fail to start and return an error, which will cause the collector to exit. The timeout defaults to 10s and can be configured with the wait_for_metadata_timeout option. example for setting the processor to wait for metadata to be synced before it is ready:

Informer Cache Resync Period

Reprocessing the informer cache periodically (resyncing) enqueues all cached K8s objects back into event handlers. In large clusters (e.g., 100K pods), this causes significant CPU spikes, memory churn, and garbage collection overhead. Because resource state modifications are already pushed immediately via Kubernetes watch events, a resync period is almost entirely unnecessary.
  • watch_sync_period (default: 5m): The resync period for K8s informers. You may set this to 0s to disable resyncing completely (recommended for large clusters).

Pod Deletion Grace Period

After receiving a pod deletion event, the processor can keep the pod’s metadata in its lookup cache for a short period before eviction. This grace window ensures that delayed spans, metrics, or logs that belong to the deleted pod can still be correctly enriched.
  • pod_delete_grace_period (default: 120s): The grace period to wait before deleting a pod’s metadata from the lookup cache after a deletion event.

Extracting attributes from pod labels and annotations

The k8sattributesprocessor can also set resource attributes from k8s labels and annotations of pods, namespaces, deployments, statefulsets, daemonsets, jobs and nodes. The config for associating the data passing through the processor (spans, metrics and logs) with specific Pod/Namespace/Deployment/StatefulSet/DaemonSet/Job/Node annotations/labels is configured via “annotations” and “labels” keys. This config represents a list of annotations/labels that are extracted from pods/namespaces/deployments/statefulsets/daemonsets/jobs/nodes and added to spans, metrics and logs. Each item is specified as a config of tag_name (representing the tag name to tag the spans with), key (representing the key used to extract value) and from (representing the kubernetes object used to extract the value). The “from” field has the following possible values: “pod”, “namespace”, “deployment”, “statefulset”, “daemonset”, “job” and “node” and defaults to “pod” if none is specified. By default, extracting metadata from Deployments, StatefulSets, DaemonSets and Jobs is disabled. Enabling extraction of these metadata comes with an extra memory consumption cost. A few examples to use this config are as follows:
The processor can be configured to set the recommended resource attributes:
  • otel_annotations will translate resource.opentelemetry.io/foo to the foo resource attribute, etc.
  • deployment_name_from_replicaset is deprecated and will be removed in future releases. Deployment names are derived from ReplicaSet names by default by trimming the pod-template-hash suffix. Set this option to false only to force ReplicaSet informer lookup for deployment names. If k8s.deployment.uid is included in the extract metadata section, or deployment labels or annotations are being extracted (i.e. any extract.labels or extract.annotations rule with from: deployment), then the Deployment/ReplicaSet informers are started and this setting is ignored. Important: You must still include k8s.deployment.name (or service.name) in the extract.metadata section for the deployment name to be extracted. The processor derives the deployment name from the ReplicaSet’s naming convention without requiring direct access to Deployment resources, but the extraction rules must be enabled. Take the following ownerReference of a pod managed by deployment for example:
The Extracted deployment name is: opentelemetry-collector. Note: When deployment names are derived from ReplicaSet names, in rare cases where deployment names are between 247 and 253 characters, Kubernetes may truncate the name in the ReplicaSet to fit the pod template hash suffix within the DNS subdomain limit (253 chars), causing the extracted k8s.deployment.name to be slightly truncated. If this affects your workloads, you can set deployment_name_from_replicaset: false or enable the k8s.deployment.uid attribute for accurate retrieval from the Kubernetes API, but at an extra cost in memory. Also note that for CronJob names (k8s.cronjob.name) a similar pattern applies, but it uses the Job informer (not ReplicaSet) and there is no deployment_name_from_replicaset-style flag. With only k8s.cronjob.name in extract.metadata, the processor derives the CronJob name from the Job’s name using a heuristic (8-digit time suffix aligned with pod creation time) and does not start a Job informer. The Job informer is started when k8s.cronjob.uid is enabled, or when labels or annotations are extracted with from: job, in which case the CronJob name can be resolved from the API when available. That reduces RBAC needs and memory use when you only need the CronJob name (no jobs watch for that attribute alone). Example:

Config example

Common Use Cases

Example 1: Basic Agent Deployment (DaemonSet)

Minimal configuration for an agent collecting telemetry from pods on the same node:
Required environment variable in your collector DaemonSet:

Example 2: Gateway Deployment with Resource Attribute Association

Gateway configuration that receives telemetry from agents that have already added pod IP:

Example 3: Production Deployment with Namespace Filtering

Configuration for monitoring a specific namespace with comprehensive metadata:

Example 4: Memory-Optimized Configuration

Minimal memory footprint configuration for large clusters:

Example 5: Multi-Container Pod Support

Configuration for extracting container-level metadata:

Role-based access control

Cluster-scoped RBAC

If you’d like to set up the k8sattributesprocessor to receive telemetry from across namespaces, it will need get, watch and list permissions on both pods and namespaces resources, for all namespaces and pods included in the configured filters. Additionally, when using k8s.deployment.uid, when using k8s.deployment.name with the deprecated deployment_name_from_replicaset: false, or when extracting labels or annotations with from: deployment, the processor needs get, watch and list permissions for replicasets resources. When using k8s.node.uid or extracting metadata from node, the processor needs get, watch and list permissions for nodes resources. With only k8s.cronjob.name (and no k8s.cronjob.uid, and no label or annotation extraction with from: job), the processor does not need get, watch and list permissions for jobs resources. When using k8s.cronjob.uid, or when extracting labels or annotations with from: job, the processor also needs get, watch and list permissions for jobs resources. Here is an example of a ClusterRole to give a ServiceAccount the necessary permissions for all pods, nodes, and namespaces in the cluster (replace <OTEL_COL_NAMESPACE> with a namespace where collector is deployed):

Namespace-scoped RBAC

When running the k8sattributesprocessor to receive telemetry traffic from pods in a specific namespace, you can use a k8s Role and Rolebinding to provide collector access to query pods and replicasets in the namespace. This would require setting the filter::namespace config as shown below.
With the namespace filter set, the processor will only look up pods and replicasets (when ReplicaSet lookup is needed, such as deployment_name_from_replicaset: false, k8s.deployment.uid, or deployment label/annotation extraction) in the selected namespace. Note that with just a role binding, the processor cannot query metadata such as labels and annotations from k8s nodes and namespaces which are cluster-scoped objects. This also means that the processor cannot set the value for k8s.cluster.uid attribute if enabled, since the k8s.cluster.uid attribute is set to the uid of the namespace kube-system which is not queryable with namespaced rbac. Please note, when extracting the workload related attributes, these workloads need to be present in the Role with the correct permissions. For example, an extraction of k8s.deployment.label.* attributes, deployments need to be present in Role. Example Role and RoleBinding to create in the namespace being watched.

Deployment scenarios

The processor can be used in collectors deployed both as an agent (Kubernetes DaemonSet) or as a gateway (Kubernetes Deployment).

As an agent

When running as an agent, the processor detects IP addresses of pods sending spans, metrics or logs to the agent and uses this information to extract metadata from pods. When running as an agent, it is important to apply a discovery filter so that the processor only discovers pods from the same host that it is running on. Not using such a filter can result in unnecessary resource usage especially on very large clusters. Once the filter is applied, each processor will only query the k8s API for pods running on its own node. Node filter can be applied by setting the filter.node config option to the name of a k8s node. While this works as expected, it cannot be used to automatically filter pods by the same node that the processor is running on in most cases as it is not know before hand which node a pod will be scheduled on. Luckily, kubernetes has a solution for this called the downward API. To automatically filter pods by the node the processor is running on, you’ll need to complete the following steps:
  1. Use the downward API to inject the node name as an environment variable. Add the following snippet under the pod env section of the OpenTelemetry container.
This will inject a new environment variable to the OpenTelemetry container with the value as the name of the node the pod was scheduled to run on.
  1. Set “filter.node_from_env_var” to the name of the environment variable holding the node name.
This will restrict each OpenTelemetry agent to query pods running on the same node only dramatically reducing resource requirements for very large clusters.

As a gateway

When running as a gateway, the processor cannot correctly detect the IP address of the pods generating the telemetry data without any of the well-known IP attributes, when it receives them from an agent instead of receiving them directly from the pods. To workaround this issue, agents deployed with the k8s_attributes processor can be configured to detect the IP addresses and forward them along with the telemetry data resources. Collector can then match this IP address with k8s pods and enrich the records with the metadata. In order to set this up, you’ll need to complete the following steps:
  1. Setup agents in passthrough mode Configure the agents’ k8s_attributes processors to run in passthrough mode.
This will ensure that the agents detect the IP address as add it as an attribute to all telemetry resources. Agents will not make any k8s API calls, do any discovery of pods or extract any metadata.
  1. Configure the collector as usual No special configuration changes are needed to be made on the collector. It’ll automatically detect the IP address of spans, logs and metrics sent by the agents as well as directly by other services/pods.

Complete Configuration Options

Below is a comprehensive configuration example with all available options:

Configuration Options Reference

Top-Level Options

Extract Options

Default metadata fields:
  • k8s.namespace.name
  • k8s.pod.name
  • k8s.pod.uid
  • k8s.pod.start_time
  • k8s.deployment.name
  • k8s.node.name
Available metadata fields: All fields listed in the “Complete Configuration Options” section above under extract.metadata.

FieldExtractConfig Options

Filter Options

FieldFilterConfig Options

PodAssociationConfig Options

AssociationSource Options

Exclude Options

ExcludePodConfig Options

Default excluded pods:
  • jaeger-agent
  • jaeger-collector

Caveats

There are some edge-cases and scenarios where k8s_attributes will not work properly.

Host networking mode

The processor cannot correct identify pods running in the host network mode and enriching telemetry data generated by such pods is not supported at the moment, unless the association rule is not based on IP attribute.

As a sidecar

The processor does not support detecting containers from the same pods when running as a sidecar. While this can be done, we think it is simpler to just use the kubernetes downward API to inject environment variables into the pods and directly use their values as tags.

Compatibility

Kubernetes Versions

This processor is tested against the Kubernetes versions specified in the e2e-tests.yml workflow. These tested versions represent the officially supported Kubernetes versions for this component.

Production Deployment Guide

Scaling Considerations

Memory Consumption

The processor maintains an in-memory cache of K8s metadata for all pods it monitors. Memory usage scales with:
  • Number of pods monitored: Each pod’s metadata (labels, annotations, owner references) is cached
  • Metadata fields extracted: More fields = more memory per pod
  • Label/annotation extraction rules: Regex patterns and multiple rules increase overhead
  • Workload metadata: Extracting deployment/statefulset/daemonset/job metadata adds additional caching
Memory estimates:
  • Agent mode (node-filtered): ~50-200 MB for 100 pods per node
  • Gateway mode (cluster-wide): ~500 MB - 2 GB for 1000-10000 pods
  • With workload metadata: Add 20-30% overhead
Optimization strategies:
  1. Use node filtering in agent deployments: filter.node_from_env_var: KUBE_NODE_NAME
  2. Limit metadata extraction: Only extract fields you need
  3. Rely on the default deployment-name heuristic: Reduces memory by not caching replicaset data
  4. Filter by namespace: Limits scope when monitoring specific applications
  5. Avoid extracting workload metadata unless necessary (deployment, statefulset, etc.)

CPU Usage

CPU usage is generally low but increases with:
  • High telemetry throughput: Each data point requires pod lookup and attribute enrichment
  • Frequent pod churn: More K8s API watch events to process
  • Complex association rules: Multiple rules with many sources
Recommended resource limits:
  • Agent mode: 100-500m CPU, 256-512 Mi memory
  • Gateway mode: 500m-2 CPU, 1-4 Gi memory

High Availability

For gateway deployments, run multiple replicas with:
  • Load balancer distributing telemetry traffic
  • Each replica independently queries K8s API and maintains its own cache
  • No shared state between replicas
  • Horizontal scaling based on CPU/memory usage
For production deployments using Helm charts, see the official OpenTelemetry Kube Stack chart and the isolated multicollector deployment example.

Graceful Shutdown

The processor is stateless and requires no special shutdown procedures:
  1. Collector receives SIGTERM
  2. Processor stops watching K8s API
  3. In-flight telemetry data is processed
  4. Collector shuts down cleanly
No persistent storage required - all metadata is refreshed from K8s API on startup.

Timestamp Format

By default, the k8s.pod.start_time uses Time.MarshalText() to format the timestamp value as an RFC3339 compliant timestamp.

Self-Observability Features

The processor exposes internal telemetry metrics for monitoring its operation. For a complete list of all available metrics, see the Internal Telemetry documentation. Key metrics to monitor:
  • otelcol_otelsvc_k8s_ip_lookup_miss: Number of times pod lookup by IP failed
    • High values indicate association issues
  • otelcol_otelsvc_k8s_pod_added / otelcol_otelsvc_k8s_pod_deleted: Track pod churn rates
    • Monitor for unexpected spikes in pod lifecycle events
  • otelcol_otelsvc_k8s_pod_table_size: Current size of pod metadata cache
    • Use to monitor memory consumption trends

Warnings

  • Memory consumption: Since the processor fetches and caches the K8s metadata for the resources of the node it is on, it consumes more memory than other processors. That consumption is compounded if users don’t filter down to only the metadata for the node the processor is running on.

Feature Gates

See documentation.md for the complete list of feature gates supported by this processor. Feature gates can be enabled using the --feature-gates flag:

Semantic Conventions Compatibility

The processor is compatible with the latest stable Semantic Conventions through the following feature gates:
  • processor.k8sattributes.DontEmitV0K8sConventions
  • processor.k8sattributes.EmitV1K8sConventions
This follows the migration pattern described by the respective Collector RFC. The breaking changes between the 2 schemas are the following:
  • container.image.tag -> container.image.tags
  • k8s.pod.labels.<key> -> k8s.pod.label.<key>
  • k8s.pod.annotations.<key> -> k8s.pod.annotation.<key>
  • k8s.node.labels.<key> -> k8s.node.label.<key>
  • k8s.node.annotations.<key> -> k8s.node.annotation.<key>
  • k8s.namespace.labels.<key> -> k8s.namespace.label.<key>
  • k8s.namespace.annotations.<key> -> k8s.namespace.annotation.<key>
All attributes emitted through the processor.k8sattributes.EmitV1K8sConventions feature gate are currently in stable stability.

Available Benchmarks

The component is tested as part of the project’s load tests, with the results being publicly available at the benchmarks page. In that page, users can find details such as memory and CPU performance when the component is used in K8s Clusters (tests use KWOK) with a range number of workloads. Refer to the test for more information about the setup.

Attributes

Resource Attributes

Configuration

Example Configuration


Last generated: 2026-08-24