> ## Documentation Index
> Fetch the complete documentation index at: https://otel.fyi/llms.txt
> Use this file to discover all available pages before exploring further.

# Prometheusremotewrite

> OpenTelemetry exporter for Prometheusremotewrite

# Prometheusremotewrite Exporter

![Status](https://img.shields.io/badge/status-beta-yellow)

**Available in:** `core`, `contrib`

**Maintainers:** [@Aneurysm9](https://github.com/Aneurysm9), [@rapphil](https://github.com/rapphil), [@dashpole](https://github.com/dashpole), [@ArthurSens](https://github.com/ArthurSens), [@ywwg](https://github.com/ywwg)

**Source:** [opentelemetry-collector-contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/prometheusremotewriteexporter)

## Supported Telemetry

![Metrics](https://img.shields.io/badge/metrics-beta-green)

## Overview

Prometheus Remote Write Exporter sends OpenTelemetry metrics
to Prometheus [remote write compatible
backends](https://prometheus.io/docs/operating/integrations/)
such as Cortex, Mimir, and Thanos.
By default, this exporter requires TLS and offers queued retry capabilities.

:warning: Non-cumulative monotonic, histogram, and summary OTLP metrics are
dropped by this exporter.

A [design doc](DESIGN.md) is available to document in detail
how this exporter works.

## Getting Started

The following settings are required:

* `endpoint` (no default): The remote write URL to send remote write samples.

By default, TLS is enabled and must be configured under `tls:`:

* `insecure` (default = `false`): whether to enable client transport security for
  the exporter's connection.

As a result, the following parameters are also required under `tls:`:

* `cert_file` (no default): path to the TLS cert to use for TLS required connections. Should
  only be used if `insecure` is set to false.
* `key_file` (no default): path to the TLS key to use for TLS required connections. Should
  only be used if `insecure` is set to false.

The following settings can be optionally configured:

* `external_labels`: map of labels names and values to be attached to each metric data point
* `headers`: additional headers attached to each HTTP request.
  * *Note the following headers cannot be changed: `Content-Encoding`, `Content-Type`, `X-Prometheus-Remote-Write-Version`, and `User-Agent`.*
* `namespace`: prefix attached to each exported metric name.
* `add_metric_suffixes`: If set to false, type and unit suffixes will not be added to metrics. Default: true. **Deprecated**: Use `translation_strategy` instead.
* `translation_strategy`: Controls how OTLP metric and attribute names are translated into Prometheus metric and label names. Options are: `UnderscoreEscapingWithSuffixes` (default), `UnderscoreEscapingWithoutSuffixes`, `NoUTF8EscapingWithSuffixes`, and `NoTranslation`. When set, this takes precedence over `add_metric_suffixes`.
* `send_metadata`: If set to true, prometheus metadata will be generated and sent. Default: false. This option is ignored when using PRW 2.0, which always includes metadata.
* `remote_write_queue`: fine tuning for queueing and sending of the outgoing remote writes.
  * `enabled`: enable the sending queue (default: `true`)
  * `queue_size`: number of OTLP metrics that can be queued. Ignored if `enabled` is `false` (default: `10000`)
  * `num_consumers`: minimum number of workers to use to fan out the outgoing requests. (default: `5` or default: `1` if `EnableMultipleWorkersFeatureGate` is enabled).
* `resource_to_telemetry_conversion`
  * `enabled` (default = false): If `enabled` is `true`, all the resource attributes will be converted to metric labels by default.
  * `exclude_service_attributes` (default = false): If set to `true`, the `service.name`, `service.instance.id` and `service.namespace`  resource attributes, which are already converted to `job` and `instance` labels respectively, will be excluded from the final metrics.
* `wal`: Write-Ahead-Log settings for the exporter.
  * `directory` (default = \`\`): The directory to store the WAL in.
  * `buffer_size` (default = `300`): Count of elements to be read from the WAL before truncating.
  * `truncate_frequency` (default = `1m`): Frequency for how often the WAL should be truncated.
  * `lag_record_frequency` (default = `15s`): Frequency for how often the exporter will record the lag of the WAL.
* `target_info`: customize `target_info` metric
  * `enabled` (default = true): If `enabled` is `true`, a `target_info` metric will be generated for each resource metric (see [https://github.com/open-telemetry/opentelemetry-specification/pull/2381](https://github.com/open-telemetry/opentelemetry-specification/pull/2381)).
* `disable_scope_info` (default = false): If `true`, the scope info labels (`otel_scope_name`, `otel_scope_version` and `otel_scope_...` attributes) will not be exported.
* `max_batch_size_bytes` (default = `3000000` -> `~2.861 mb`): Maximum size of a batch of samples to be sent to the remote
  write endpoint. If the batch size is larger than this value, it will be split into multiple batches. This option is ignored
  when using the wal and where the wal buffer\_size / truncate\_frequency will be used.
* `max_batch_request_parallelism` (default = `5`): Maximum parallelism allowed when sending multiple requests to the remote write endpoint.
  If the remote write endpoint does not support out of order samples, this should be set to `1`.
* `protobuf_message` (default = `prometheus.WriteRequest`):
  * Protobuf message to use when writing to the remote write endpoint. This option is ignored unless the `exporter.prometheusremotewritexporter.enableSendingRW2` feature gate is enabled.
  * `prometheus.WriteRequest` is the message used in [Remote Write 1.0](https://prometheus.io/docs/specs/remote_write_spec/).
  * `io.prometheus.write.v2.Request` is the message used in [Remote Write 2.0](https://prometheus.io/docs/specs/remote_write_spec_2_0/). It is more efficient, always includes metadata, and adds support for the created timestamp and native histograms. Your remote storage provider must support PRW 2.0 to be able to use this message. PRW 2.0 support is currently **In Development** and is only partially implemented, thus, not ready for usage.

Example:

```yaml theme={null}
exporters:
  prometheus_remote_write:
    endpoint: "https://my-cortex:7900/api/v1/push"
    wal: # Enabling the Write-Ahead-Log for the exporter.
      directory: ./prom_rw # The directory to store the WAL in
      buffer_size: 100 # Optional count of elements to be read from the WAL before truncating; default of 300
      truncate_frequency: 45s # Optional frequency for how often the WAL should be truncated. It is a time.ParseDuration; default of 1m
    resource_to_telemetry_conversion:
      enabled: true # Convert resource attributes to metric labels
```

Example:

```yaml theme={null}
exporters:
  prometheus_remote_write:
    endpoint: "https://my-cortex:7900/api/v1/push"
    external_labels:
      label_name1: label_value1
      label_name2: label_value2
```

> \[!NOTE]
> The deprecated coponent type `prometheusremotewrite` (without the underscores) can still be used as an alias and will log a deprecation warning.

## Advanced Configuration

Several helper files are leveraged to provide additional capabilities automatically:

* [HTTP settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/confighttp/README.md), note that the exporter only supports `snappy` compression type as it's [required](https://prometheus.io/docs/specs/remote_write_spec/#protocol) by the Prometheus remote write protocol.
* [TLS and mTLS settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md)
* [Retry and timeout settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md), note that the exporter doesn't support `sending_queue` but provides `remote_write_queue`.

### Feature gates

#### RetryOn429

This exporter has feature gate: `exporter.prometheusremotewritexporter.RetryOn429`.
When this feature gate is enable the prometheus remote write exporter will retry on 429 http status code with the provided retry configuration.
It currently doesn't support respecting the http header `Retry-After` if provided since the retry library used doesn't support this feature.

To enable it run collector with enabled feature gate `exporter.prometheusremotewritexporter.RetryOn429`. This can be done by executing it with one additional parameter - `--feature-gates=telemetry.useOtelForInternalMetrics`.

#### EnableMultipleWorkersFeatureGate

This exporter has feature gate: `+exporter.prometheusremotewritexporter.EnableMultipleWorkers`.

When this feature gate is enabled, `num_consumers` will be used as the worker counter for handling batches from the queue, and `max_batch_request_parallelism` will be used for parallelism on single batch bigger than `max_batch_size_bytes`.
Enabling this feature gate, with `num_consumers` higher than 1 requires the target destination to supports ingestion of OutOfOrder samples. See [Multiple Consumers and OutOfOrder](#multiple-consumers-and-outoforder) for more info

## Metric names and labels normalization

OpenTelemetry metric names and attributes are normalized to be compliant with Prometheus naming rules. [Details on this normalization process are described in the Prometheus translator module](../../pkg/translator/prometheus/).

## Setting resource attributes as metric labels

By default, resource attributes are added to a special metric called `target_info`. To select and group by metrics by resource attributes, you [need to do join on `target_info`](https://prometheus.io/docs/prometheus/latest/querying/operators/#many-to-one-and-one-to-many-vector-matches). For example, to select metrics with `k8s_namespace_name` attribute equal to `my-namespace`:

```promql theme={null}
app_ads_ad_requests_total * on (job, instance) group_left target_info{k8s_namespace_name="my-namespace"}
```

Or to group by a particular attribute (for ex. `k8s_namespace_name`):

```promql theme={null}
sum by (k8s_namespace_name) (app_ads_ad_requests_total * on (job, instance) group_left(k8s_namespace_name) target_info)
```

This is not a common pattern, and we recommend copying the most common resource attributes into metric labels. You can do this through the transform processor:

```yaml theme={null}
processor:
  transform:
    metric_statements:
      - context: datapoint
        statements:
        - set(attributes["namespace"], resource.attributes["k8s.namespace.name"])
        - set(attributes["container"], resource.attributes["k8s.container.name"])
        - set(attributes["pod"], resource.attributes["k8s.pod.name"])
```

After this, grouping or selecting becomes as simple as:

```promql theme={null}
app_ads_ad_requests_total{namespace="my-namespace"}

sum by (namespace) (app_ads_ad_requests_total)
```

[beta]: https://github.com/open-telemetry/opentelemetry-collector#beta

[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib

[core]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol

## Multiple Consumers and OutOfOrder

**DISCLAIMER**: This snippet applies only to Prometheus, other remote write destinations using Prometheus Protocol (ex: Thanos/Grafana Mimir/VictoriaMetrics) may have different settings.

By default, Prometheus expects samples to be ingested sequentially, in temporal order.

When multiple consumers are enabled, the temporal ordering of the samples written to the target destination is not deterministic, and temporal ordering can no longer be guaranteed. For example, one worker may push a sample for `t+30s`, and a second worker may push an additional sample but for `t+15s`.

Vanilla Prometheus configurations will reject these unordered samples and you'll receive "out of order" errors.

Out-of-order support in Prometheus must be enabled for multiple consumers.
This can be done by using the `tsdb.out_of_order_time_window: 10m` settings. Please choose an appropriate time window to support pushing the worst-case scenarios of a "queue" build-up on the sender side.

See for more info:

* [https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tsdb](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tsdb)

## Configuration

### Example Configuration

```yaml theme={null}
prometheus_remote_write:

prometheus_remote_write/2:
  namespace: "test-space"
  max_batch_request_parallelism: 10
  retry_on_failure:
    enabled: true
    initial_interval: 10s
    max_interval: 60s
    max_elapsed_time: 10m
  endpoint: "localhost:8888"
  tls:
    ca_file: "/var/lib/mycert.pem"
  write_buffer_size: 524288
  add_metric_suffixes: false
  headers:
    Prometheus-Remote-Write-Version: "0.1.0"
    X-Scope-OrgID: "234"
  external_labels:
    key1: value1
    key2: value2
  resource_to_telemetry_conversion:
    enabled: true
  remote_write_queue:
    queue_size: 2000
    num_consumers: 10

prometheus_remote_write/negative_queue_size:
  endpoint: "localhost:8888"
  remote_write_queue:
    queue_size: -1
    num_consumers: 10

prometheus_remote_write/negative_num_consumers:
  endpoint: "localhost:8888"
  remote_write_queue:
    queue_size: 5
    num_consumers: -1

prometheus_remote_write/less_than_1_max_batch_request_parallelism:
  endpoint: "localhost:8888"
  max_batch_request_parallelism: 0

prometheus_remote_write/disabled_target_info:
  endpoint: "localhost:8888"
  target_info:
    enabled: false

prometheus_remote_write/disabled_queue:
  endpoint: "localhost:8888"
  remote_write_queue:
    enabled: false
    num_consumers: 10

prometheus_remote_write/non_snappy_compression_type:
  endpoint: "localhost:8888"
  compression: "gzip"

prometheus_remote_write/unknown_protobuf_message:
  protobuf_message: "io.prometheus.write.v4.Request"

prometheus_remote_write/translation_strategy:
  endpoint: "localhost:8888"
  protobuf_message: "io.prometheus.write.v2.Request"
  translation_strategy: "NoTranslation"

prometheus_remote_write/invalid_translation_strategy:
  endpoint: "localhost:8888"
  translation_strategy: "invalid_strategy"

prometheus_remote_write/v1_no_utf8:
  endpoint: "localhost:8888"
  protobuf_message: "prometheus.WriteRequest"
  translation_strategy: "NoUTF8EscapingWithSuffixes"

prometheus_remote_write/v1_no_translation:
  endpoint: "localhost:8888"
  protobuf_message: "prometheus.WriteRequest"
  translation_strategy: "NoTranslation"
```

***

*Last generated: 2026-07-06*