> ## Documentation Index
> Fetch the complete documentation index at: https://otel.fyi/llms.txt
> Use this file to discover all available pages before exploring further.

# Dataset

> OpenTelemetry exporter for Dataset

# Dataset Exporter

![Status](https://img.shields.io/badge/status-alpha-red)

**Available in:** `contrib`

**Maintainers:** [@atoulme](https://github.com/atoulme), [@martin-majlis-s1](https://github.com/martin-majlis-s1), [@zdaratom-s1](https://github.com/zdaratom-s1), [@tomaz-s1](https://github.com/tomaz-s1)

**Source:** [opentelemetry-collector-contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/datasetexporter)

## Supported Telemetry

![Logs](https://img.shields.io/badge/logs-alpha-blue) ![Traces](https://img.shields.io/badge/traces-alpha-orange)

## Overview

This exporter sends logs to [DataSet](https://www.dataset.com/).

See the [Getting Started](https://app.scalyr.com/help/getting-started) guide.

## Configuration

### Required Settings

* `dataset_url` (no default): The URL of the DataSet API that ingests the data. Most likely [https://app.scalyr.com](https://app.scalyr.com).
* `api_key` (no default): The "Log Write" API Key required to use API. Instructions how to get [API key](https://app.scalyr.com/help/api-keys).

If you do not want to specify `api_key` in the file, you can use the [builtin functionality](https://opentelemetry.io/docs/collector/configuration/#configuration-environment-variables) and use `api_key: ${env:DATASET_API_KEY}`.

### Server Host Settings

Specifying the server host is crucial for ensuring the correct functionality of DataSet.
DataSet expects the server host value to be provided in the `serverHost` attribute.
If the server host value is stored in a different attribute, you can use the [resourceprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/resourceprocessor/README.md) or [attributesprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/attributesprocessor) to copy it into the `serverHost` attribute.

You can also utilize the `server_host` settings (described below) to populate the serverHost attribute with different values.

The process of populating the serverHost attribute works as follows:

* If the `serverHost` attribute is specified and not empty in the log or trace, then it is used.
* If the `serverHost` attribute is specified and not empty in the [resource](https://opentelemetry.io/docs/specs/otel/resource/sdk/), then it is used.
* If the `host.name` attribute is specified and not empty in the [resource](https://opentelemetry.io/docs/specs/otel/resource/sdk/), then it is used.
* If the `server_host.server_host` setting is specified and not empty, then it is used.
* If `server_host.use_host_name` setting is set to `true`, the `hostname` of the node is used.

Make sure to provide the appropriate server host value in the `serverHost` attribute to ensure the proper functionality of DataSet and accurate handling of events.

### Optional Settings

* `debug` (default = false): Adds `session_key` to the server fields. It's useful for debugging throughput issues.
* `buffer`:
  * `max_lifetime` (default = 5s): The maximum delay between sending batches from the same session.
  * `purge_older_than` (default = 30s): The maximum delay between receiving data for the same session after which resources associated with it are purged.
  * `group_by` (default = \[]): The list of attributes based on which events should be grouped. They are moved from the event attributes to the session info and shown as server fields in the UI.
  * `retry_initial_interval` (default = 5s): Time to wait after the first failure before retrying.
  * `retry_max_interval` (default = 30s): Is the upper bound on backoff.
  * `retry_max_elapsed_time` (default = 300s): Is the maximum amount of time spent trying to send a buffer.
  * `retry_shutdown_timeout` (default = 30s): The maximum time for which it will try to send data to the DataSet during shutdown. This value should be shorter than container's grace period.
  * `max_parallel_outgoing` (default = 100): The maximum number of parallel outgoing requests.
* `logs`:
  * `export_resource_info_on_event` (default = false): Include LogRecord resource information (if available) on the DataSet event.
  * `export_resource_prefix` (default = 'resource.attributes.'): A prefix string for the resource, if `export_resource_info_on_event` is enabled.
  * `export_scope_info_on_event` (default = true): Include LogRecord scope information (if available) on the DataSet event.
  * `export_scope_prefix` (default = 'scope.attributes.'):  A prefix string for the scope, if `export_scope_info_on_event` is enabled.
  * `export_separator` (default = '.'): The separator to add between keys when flattening nested structures (maps, arrays).
  * `export_distinguishing_suffix` (default = '\_'): A suffix string to resolve naming collisions when flattening.
  * `decompose_complex_message_field` (default = false): Decompose complex body / message field types (e.g. a maps, arrays) into separate fields.
  * `decomposed_complex_message_prefix` (default = 'body.map.'): A prefix string to use when a complex message is decomposed.
* `traces`:
  * `export_separator` (default = '.'): The separator to add between keys when flattening nested structures (maps, arrays).
  * `export_distinguishing_suffix` (default = '\_'): A suffix string to resolve naming collisions when flattening.
* `server_host`:
  * `server_host` (default = ''): Specifies the server host to be used for the events.
  * `use_hostname` (default = true): Determines whether the `hostname` of the node should be used as the server host for the events. When set to `true`, the node's `hostname` is automatically used.
* `retry_on_failure`: See [retry\_on\_failure](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md)
* `sending_queue`: See [sending\_queue](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md)
* `timeout`: See [timeout](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md)

### Attributes

Enabled attributes are exported in the order:

1. Log properties
2. Body
3. Resource attributes
4. Scope attributes
5. Log attributes

If there is a name conflict, the `export_distinguishing_suffix` value is appended to the later attribute's name. If the `export_distinguishing_suffix` value is an empty string, then the value from the last attribute is used.

#### Example

Example LogRecord:

```
Log
- body:
  - b: 1
  - x: "b"
- resource:
  - r: 2
  - x: "r"
- scope:
  - s: 3
  - x: "s"
- attribute:
  - a: 4
  - x: "a"
  - map:
    - m1: 5
    - m2: 6
```

Then the event will look like:

* Default settings for `logs`:
  * Event:
    ```
    - message: "{\"b\": 1, \"x\": \"b\"}"
    - scope.attributes.s: 3
    - scope.attributes.x: "s"
    - a: 4
    - x: "a"
    - map.m1: 5
    - map.m2: 6
    ```
* Everything enabled:
  * Configuration:
    ```
      logs:
        export_resource_info_on_event: true
        export_resource_prefix: "r."
        export_scope_info_on_event: true
        export_scope_prefix: "s."
        decompose_complex_message_field: true
        decomposed_complex_message_prefix: "m."
        export_separator: "-"
        export_distinguishing_suffix: "_"
    ```
  * Event:
    ```
    - message: "{\"b\": 1, \"x\": \"b\"}"
    - m.b: 1
    - m.x: "b"
    - r.r: 2
    - r.x: "r"
    - s.s: 3
    - s.x: "s"
    - a: 4
    - x: "a"
    - map-m1: 5
    - map-m2: 6
    ```
* Everything enabled, prefixes are empty strings:
  * Configuration:
    ```
      logs:
        export_resource_info_on_event: true
        export_resource_prefix: ""
        export_scope_info_on_event: true
        export_scope_prefix: ""
        decompose_complex_message_field: true
        decomposed_complex_message_prefix: ""
        export_separator: "-"
        export_distinguishing_suffix: "_"
    ```
  * Event:
    ```
    - message: "{\"b\": 1, \"x\": \"b\"}"
    - b: 1
    - x: "b"
    - r: 2
    - x_: "r"
    - s: 3
    - x__: "s"
    - a: 4
    - x___: "a"
    - map-m1: 5
    - map-m2: 6
    ```
* Everything enabled, prefixes are empty strings, suffix is empty string:
  * Configuration:
    ```
      logs:
        export_resource_info_on_event: true
        export_resource_prefix: ""
        export_scope_info_on_event: true
        export_scope_prefix: ""
        decompose_complex_message_field: true
        decomposed_complex_message_prefix: ""
        export_separator: "-"
        export_distinguishing_suffix: ""
    ```
  * Event:
    ```
    - message: "{\"b\": 1, \"x\": \"b\"}"
    - b: 1
    - r: 2
    - s: 3
    - a: 4
    - x: "a"
    - map-m1: 5
    - map-m2: 6
    ```

Field names can have `.` dots, `_` underscores, and `-` hyphens. You must escape slashes in Search and PowerQueries. For example, search the field name `app.kubernetes.io/component` as `app.kubernetes.io\/component`.

### Example

```yaml theme={null}
processors:
  attributes:
    - key: serverHost
      action: insert
      from_attribute: container_id
  resource:
    attributes:
      - key: serverHost
        from_attribute: node_id
        action: insert      

exporters:
  dataset/logs:
    # DataSet API URL, https://app.eu.scalyr.com for DataSet EU instance
    dataset_url: https://app.scalyr.com
    # API Key
    api_key: your_api_key
    buffer:
      # Send buffer to the API at least every 5s
      max_lifetime: 5s
      # Group data based on these attributes
      group_by:
        - container_id
      # try to send data to the DataSet for at most 30s during shutdown
      retry_shutdown_timeout: 30s
    server_host:
      # If the serverHost attribute is not specified or empty,
      # use the value from the env variable SERVER_HOST
      server_host: ${env:SERVER_HOST}
      # If server_host is not set, use the hostname value
      use_hostname: true
    sending_queue:
      # Enable batching
      batch:

  dataset/traces:
    # DataSet API URL, https://app.eu.scalyr.com for DataSet EU instance
    dataset_url: https://app.scalyr.com
    # API Key
    api_key: your_api_key
    buffer:
      max_lifetime: 15s
      group_by:
        - resource_service.instance.id
    sending_queue:
      # Enable batching
      batch:

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [attributes]
      # add dataset among your exporters
      exporters: [dataset/logs]
    traces:
      receivers: [otlp]
      # add dataset among your exporters
      exporters: [dataset/traces]
```

### Handling `serverHost` Attribute

Based on the given configuration and scenarios, here's the expected behavior:

1. Resource: `{'node_id:' 'node-pay-01', 'host.name': 'host-pay-01'}`, Log: `{'container_id': 'cont-pay-01'}`, Env: `SERVER_HOST='server-pay-01'`, Hostname: `ip-172-31-27-19`
   * Since the attribute `container_id` is set, `attributesprocessor` will copy this value to the `serverHost`.
   * Used `serverHost` will be `cont-pay-01`.
2. Resource: `{'node_id': 'node-pay-01', 'host.name': 'host-pay-01'}`, Log: `{'attribute.foo': 'Bar'}`, Env: `SERVER_HOST='server-pay-01'`, Hostname: `ip-172-31-27-19`
   * Since the resource attribute `node_id` is set, `resourceprocessor` will copy this value to the `serverHost`.
   * Used `serverHost` will be `node-pay-01`.
3. Resource: `{'host.name': 'host-pay-01'}`, Log: `{'attribute.foo': 'Bar'}`, Env: `SERVER_HOST='server-pay-01'`, Hostname: `ip-172-31-27-19`
   * Since the resource attribute `host.name` is set, it will be used.
   * Used `serverHost` will be `host-pay-01`.
4. Resource: `{}`, Log: `{'attribute.foo': 'Bar'}`, Env: `SERVER_HOST='server-pay-01'`, Hostname: `ip-172-31-27-19`
   * Since the attribute `container_id` is not set, the value from the environmental variable `SERVER_HOST`  will be copied to the `serverHost`.
   * Used `serverHost` will be `server-pay-01`.
5. Resource: `{}`, Log: `{'attribute.foo': 'Bar'}`, Env: `SERVER_HOST=''`, Hostname: `ip-172-31-27-19`
   * Since the attribute `container_id` is not set and the environmental variable `SERVER_HOST` is empty, the `hostname` of the node (`ip-172-31-27-19`) will be used as the fallback value for `serverHost`.
   * Used `serverHost` will be `ip-172-31-27-19`.

## Metrics

To enable metrics you have to:

1. Run collector with enabled feature gate `telemetry.useOtelForInternalMetrics`. This can be done by executing it with one additional parameter - `--feature-gates=telemetry.useOtelForInternalMetrics`.
2. Enable metrics scraping as part of the configuration and add receiver into services:
   ```yaml theme={null}
   receivers:
     prometheus:
       config:
         scrape_configs:
           - job_name: 'otel-collector'
             scrape_interval: 5s
             static_configs:
               - targets: ['0.0.0.0:8888']
   ...
   service:
     pipelines:
       metrics:
         # add prometheus among metrics receivers
         receivers: [prometheus]
         exporters: [otlp_http/prometheus, debug]
   ```

### Available Metrics

Available metrics contain `dataset` in their name. There are counters related to the
number of processed events (`events`), buffers (`buffer`), sessions (`sessions`), and transferred bytes (`bytes`).
There are also histograms related to response times (`responseTime`) and payload size (`payloadSize`).

There are several counters related to events/buffers:

* `enqueued` - the number of received entities
* `processed` - the number of entities that were accepted by the next layer
* `dropped` - the number of entities that were not accepted by the next layer
* `broken` - the number of entities that were somehow corrupted during processing (should be 0)

The number of entities, that are still in the queue can be computed as `enqueued - (processed + dropped + broken)`.

## Configuration

### Example Configuration

```yaml theme={null}
dataset/minimal:
  dataset_url: https://app.scalyr.com
  api_key: key-minimal

dataset/lib:
  dataset_url: https://app.eu.scalyr.com
  api_key: key-lib
  buffer:
    max_lifetime: 345ms
    group_by:
      - attributes.container_id
      - attributes.log.file.path

dataset/full:
  dataset_url: https://app.scalyr.com
  api_key: key-full
  debug: true
  buffer:
    max_lifetime: 3456ms
    purge_older_than: 78s
    group_by:
      - body.map.kubernetes.pod_id
      - body.map.kubernetes.docker_id
      - body.map.stream
    retry_initial_interval: 21s
    retry_max_interval: 22s
    retry_max_elapsed_time: 23s
    retry_shutdown_timeout: 24s
    max_parallel_outgoing: 25
  logs:
    export_resource_info_on_event: true
    export_resource_prefix: "_resource_"
    export_scope_info_on_event: true
    export_scope_prefix: "_scope_"
    export_separator: "_X_"
    export_distinguishing_suffix: "_L_"
    decompose_complex_message_field: true
    decomposed_complex_message_prefix: "_body_"
  traces:
    export_separator: "_Y_"
    export_distinguishing_suffix: "_T_"
  server_host:
    use_hostname: false
    server_host: "server-host"
  retry_on_failure:
    enabled: true
    initial_interval: 11
    randomization_factor: 0.113
    multiplier: 11.6
    max_interval: 12
    max_elapsed_time: 13
  sending_queue:
    enabled: true
    num_consumers: 14
    queue_size: 15
  timeout:
    timeout: 16
```

***

*Last generated: 2026-07-06*