> ## Documentation Index
> Fetch the complete documentation index at: https://otel.fyi/llms.txt
> Use this file to discover all available pages before exploring further.

# Otelarrow

> OpenTelemetry exporter for Otelarrow

# Otelarrow Exporter

![Status](https://img.shields.io/badge/status-beta-yellow)

**Available in:** `contrib`, `k8s`

**Maintainers:** [@jmacd](https://github.com/jmacd), [@JakeDern](https://github.com/JakeDern)

**Source:** [opentelemetry-collector-contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/otelarrowexporter)

## Supported Telemetry

![Logs](https://img.shields.io/badge/logs-beta-blue) ![Metrics](https://img.shields.io/badge/metrics-beta-green) ![Traces](https://img.shields.io/badge/traces-beta-orange)

## Overview

Exports telemetry data using [OpenTelemetry Protocol with Apache
Arrow](https://github.com/open-telemetry/otel-arrow) components with
support for both OpenTelemetry Protocol with Apache
Arrow and standard [OpenTelemetry Protocol](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/otlp.md)
(OTLP) protocol via gRPC.

## Getting Started

The [OpenTelemetry Protocol with Apache
Arrow](https://github.com/open-telemetry/otel-arrow) exporter combines
the features and configuration syntax of the core OpenTelemetry
Collector [OTLP
exporter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlpexporter)
component with additional support for the OpenTelemetry Protocol with
Apache Arrow.

OpenTelemetry Protocol with Apache Arrow supports column-oriented data
transport using the Apache Arrow data format.  This component converts
OTLP data into an optimized representation and then sends batches of
data using Apache Arrow to encode the stream.  The [OpenTelemetry
Protocol with Apache Arrow receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/otelarrowreceiver) component contains logic to reverse the process used in this
component.

The use of an OpenTelemetry Protocol with Apache Arrow
exporter-receiver pair is recommended when the network is expensive.
Typically, expect to see a 50% reduction in bandwidth compared with
the same data being sent using standard OTLP/gRPC with Zstd
compression, batch sizes being equal.

This component includes all the features and configuration of the core
OTLP exporter, making it possible to upgrade from the core OTLP
exporter component.  This is as simple as replacing "otlp" with
"otelarrow" as the component name in the collector configuration.

To enable the OpenTelemetry Protocol with Apache Arrow exporter,
include it in the list of exporters for a pipeline.  The `endpoint`
setting is required.  The `tls` setting is required for insecure
transport.

* `endpoint` (no default): host:port to which the exporter is going to send OTLP trace data,
  using the gRPC protocol. The valid syntax is described in
  [grpc's naming.md](https://github.com/grpc/grpc/blob/master/doc/naming.md).
  If a scheme of `https` is used then client transport security is enabled and overrides the `insecure` setting.
* `tls`: see [TLS Configuration Settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md) for the full set of available options.

Example:

```yaml theme={null}
exporters:
  otelarrow/secure:
    endpoint: external-collector:4317
    tls:
      cert_file: file.cert
      key_file: file.key
  otelarrow/insecure:
    endpoint: internal-collector:4317
    tls:
      insecure: true
```

By default, `zstd` compression is enabled at the gRPC level.  See
[compression configuration](#compression-configuration) below.  To
disable gRPC-level compression, configure "none":

```yaml theme={null}
exporters:
  otelarrow:
    compression: none
    endpoint: ...
    tls: ...
```

## Configuration

Several helper files are leveraged to provide additional capabilities automatically:

* [gRPC settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configgrpc/README.md)
* [TLS and mTLS settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md)
* [Queuing, retry and timeout settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md)

### Arrow-specific Configuration

In the `arrow` configuration block, the following settings enable and
disable the use of OpenTelemetry Protocol with Apache Arrow as opposed
to standard OTLP.

* `disabled` (default: false): disables use of Arrow, causing the exporter to use standard OTLP
* `disable_downgrade` (default: false): prevents this exporter from using standard OTLP.

The following setting determines how long a stream will stay open.
Stream lifetime is limited to 30 seconds because compression benefit
is limited at that point and shorter streams make load balancing
easier.

* `max_stream_lifetime` (default: 30s): duration after which streams
  are recycled.

The following setting determines memory and CPU resources that the
exporter will use:

* `num_streams` (default: `max(1, NumCPU()/2)`): the number of concurrent Arrow streams

The `num_streams` default limits the exporter stream count to half the
number of CPUs or 1, whichever is greater.  When `num_streams` is
greater than one, a configurable policy determines how load is
assigned across streams to balance load.  The supported policies are
`leastloaded`, which picks the stream with the smallest number of
outstanding requests, and `leastloadedN` for `N <= num_streams`, which
limits the decision to a random subset of `N` streams.

* `prioritizer` (default: "leastloaded"): policy for distributing load across multiple streams.

### Matching Metadata Per Stream

The following configuration values allow for separate streams per unique
metadata combinations:

* `metadata_keys` (default = empty): When set, this exporter will create one
  arrow exporter instance per distinct combination of values in the
  client.Metadata.
* `metadata_cardinality_limit` (default = 1000): When metadata\_keys is not empty,
  this setting limits the number of unique combinations of metadata key values
  that will be processed over the lifetime of the exporter.

### Network Configuration

This component uses `round_robin` by default as the gRPC load
balancer.  This can be modified using the `balancer_name` setting, for
example, to configure the `pick_first` balancer:

```yaml theme={null}
exporters:
  otelarrow:
    balancer_name: pick_first
    endpoint: ...
    tls: ...
```

When the server or an intermediate proxy uses a keepalive setting, the
Arrow-specific `max_stream_lifetime` setting is critical to avoiding
abrupt termination of Arrow streams, which causes retries of the
in-flight requests.  The maximum stream lifetime should be set to a
value less than the minimum of the server's keepalive parameter (and
any of the intermediate proxies), plus the export timeout.

```yaml theme={null}
exporters:
  otelarrow:
    timeout: 30s
    arrow:
      max_stream_lifetime: 9m30s
    endpoint: ...
    tls: ...
```

When this is configured, the stream will terminate cleanly without
causing retries, with `OK` gRPC status.

The [corresponding `otelarrowreceiver` keepalive setting](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/otelarrowreceiver#keepalive-configuration), that is
compatible with the one above, reads:

```
receivers:
  otelarrow:
    protocols:
      grpc:
        keepalive:
          server_parameters:
            max_connection_age: 1m
            max_connection_age_grace: 10m
```

### Exporter metrics

In addition to the standard
[exporterhelper](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md)
and
[obsreport](https://pkg.go.dev/go.opentelemetry.io/collector/obsreport)
metrics, this component provides network-level measurement instruments
which we anticipate will become part of `exporterhelper` and/or
`obsreport` in the future.  At the `normal` level of metrics detail:

* `otelcol_exporter_sent`: uncompressed bytes sent, prior to compression
* `otelcol_exporter_sent_wire`: compressed bytes sent, on the wire.

Arrow's compression performance can be derived by dividing the average
`otelcol_exporter_sent` value by the average `otelcol_exporter_sent_wire` value.

At the `detailed` metrics detail level, information about the stream
of data being returned to the exporter will be instrumented:

* `otelcol_exporter_recv`: uncompressed bytes received, prior to compression
* `otelcol_exporter_recv_wire`: compressed bytes received, on the wire.

### Compression Configuration

The exporter supports configuring Zstd compression at both the gRPC
and the Arrow level.  The exporter metrics described above will be
correct in either case.  The default settings are subject to change as
we gain experience.

See the Collector [compression
comparison](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configgrpc/README.md#compression-comparison)
for general information about the choice of Zstd by default, for other
general compression configuration and benchmark information.

For the OpenTelemetry Protocol with Apache Arrow streams specifically,
gRPC-level the Zstd compression level can be configured.  However,
there is an important caveat: the gRPC-Go library requires that
compressor implementations be registered statically.  These libraries
use compressors named `zstdarrow1`, `zstdarrow2`, ..., `zstdarrow10`,
supporting 10 configurable compression levels.  Note, however that
these configurations are static and only one unique configuration is
possible per level.  It is possible to configure multiple
OpenTelemetry Protocol with Apache Arrow exporters with different Zstd
configuration simply by using distinct levels.

Under `arrow`, the `zstd` sub-configuration has the following fields:

* `level`: in the range 1-10 determines a number of defaults (default 5)
* `window_size_mib`: size of the Zstd window in MiB, 0 indicates to determine based on level (default 0)
* `concurrency`: controls background CPU used for compression, 0 indicates to let `zstd` library decide (default 1)

The exporter supports configuring compression at the [Arrow
columnar-protocol
level](https://arrow.apache.org/docs/format/Columnar.html#format-ipc).

* `payload_compression` (default "zstd"): compression applied at the Arrow IPC level.

Compression at the Arrow level is enabled by default because it boosts
compression slightly and helps Arrow payloads meet gRPC maximum
request size limits.  Compression settings at the Arrow IPC level
cannot be further configured.

For example, two exporters may be configured with multiple zstd
configurations, provided they use different levels:

```yaml theme={null}
exporters:
  otelarrow/best:
    compression: zstd  # describes gRPC-level compression (default "zstd")
    arrow:
      zstd:
        level: 10      # describes gRPC-level compression level (default 5)
  otelarrow/fastest:
    compression: zstd
    arrow:
      zstd:
        level: 1       # 1 is the "fastest" compression level
```

### Batching Configuration

This exporter supports built-in `exporterhelper` support for combined
queue and batch behavior via the `sending_queue` settings. Note that
the bytes-based batching is supported, but that the exporterhelper
estimates batch sizes using the OTLP representation, not considering
Arrow compression.

In the default configuration, without a persistent storage extension,
the exporter uses an in-memory queue and will respond to the caller
before the export completes.

In the `sending_queue` structure, the default settings set by this
component are:

* `block_on_overflow: true`
* `wait_for_result: false`
* `sizer: items`
* `queue_size: 100_000`
* `batch::flush_timeout: 1s`
* `batch::min_size: 1_000`
* `batch::max_size: 1_500`
* `num_consumers: 100`

This indicates to use the in-memory queue, to return success to the
client on acceptance and block when full, to allow 100 thousand items
to export concurrently in up to 100 concurrent batches of 1000 to 1500
items each.

For additional safety in the event of a Collector crash, set
`wait_for_result: true`. The example below demonstrates how to use the
in-memory queue for batching with this additional level of safety:

```
exporters:
  otelarrow:
    # ...
    sending_queue:
      enabled: true

          # Use wait_for_result: true for additional safety, otherwise
          # a collector crash will cause loss of data.
          wait_for_result: true

          # This is the default setting, it ensures the exporter will
          # block the pipeline (subject to deadline) instead of failing
          # fast when overflow data arrives.
          block_on_overflow: true

          # The queue will admit 1 million items into the queue and return
          # success before blocking new requests.
          sizer: items
          queue_size: 1_000_000

          # Use relatively large batches, improves compression.
      batch:
            flush_timeout: 1s
            min_size: 4_000
            max_size: 5_000

      # With max-size batches, we need 200 consumers to keep the
          # OTel-Arrow streams busy. There will be (num_consumers /
          # num_streams) pending requests per stream on average.
          num_consumers: 200

          # Optional persistent storage. If this is set, you
          # can safely use wait_for_result: false above.
          # storage: name_of_extension
```

## Configuration

### Example Configuration

```yaml theme={null}
endpoint: "1.2.3.4:1234"
compression: "none"
tls:
  ca_file: /var/lib/mycert.pem
timeout: 10s
sending_queue:
  enabled: true
  num_consumers: 2
  queue_size: 10
  batch:
    flush_timeout: 200ms
    min_size: 1000
    max_size: 10000
retry_on_failure:
  enabled: true
  initial_interval: 10s
  randomization_factor: 0.7
  multiplier: 1.3
  max_interval: 60s
  max_elapsed_time: 10m
auth:
  authenticator: nop
headers:
  "can you have a . here?": "F0000000-0000-0000-0000-000000000000"
  header1: "234"
  another: "somevalue"
keepalive:
  time: 20s
  timeout: 30s
  permit_without_stream: true
balancer_name: "experimental"
arrow:
  num_streams: 2
  disabled: false
  max_stream_lifetime: 2h
  payload_compression: "zstd"
  prioritizer: leastloaded8
```

***

*Last generated: 2026-07-06*