​

Elasticsearch Exporter

​

Supported Telemetry

​

Overview

​

Configuration options

• endpoint (no default): The target Elasticsearch URL to which data will be sent (e.g. https://elasticsearch:9200)
• endpoints (no default): A list of Elasticsearch URLs to which data will be sent, attempted in round-robin order
• cloudid (no default): The Elastic Cloud ID of the Elastic Cloud Cluster to which data will be sent (e.g. foo:YmFyLmNsb3VkLmVzLmlvJGFiYzEyMyRkZWY0NTY=)

• user (optional): Username used for HTTP Basic Authentication.
• password (optional): Password used for HTTP Basic Authentication.
• api_key (optional): Elasticsearch API Key in “encoded” format (e.g. VFR2WU41VUJIbG9SbGJUdVFrMFk6NVVhVDE3SDlSQS0wM1Rxb24xdXFldw==).

exporters:
  elasticsearch:
    endpoint: https://elastic.example.com:9200
    auth:
      authenticator: basicauth

extensions:
  basicauth:
    client_auth:
      username: elastic
      password: changeme

······

service:
  extensions: [basicauth]
  pipelines:
    logs:
      receivers: [otlp]
      exporters: [elasticsearch]
    traces:
      receivers: [otlp]
      exporters: [elasticsearch]

​

Advanced configuration

​

HTTP settings

​

Queuing and batching

sending_queue:
  enabled: true
  sizer: requests
  num_consumers: 10
  queue_size: 10
  batch:
    flush_timeout: 10s
    min_size: 1e+6 // 1MB
    max_size: 5e+6 // 5MB
    sizer: bytes

​

Elasticsearch document routing

1. “Static mode”: Route to logs_index for log records, metrics_index for data points and traces_index for spans, if these configs are not empty respectively. ¹
2. “Dynamic - Index attribute mode”: Route to index name specified in elasticsearch.index attribute (precedence: log record / data point / span attribute > scope attribute > resource attribute) if the attribute exists. ¹
3. “Dynamic - Data stream routing mode”: Route to data stream constructed from ${data_stream.type}-${data_stream.dataset}-${data_stream.namespace}, where data_stream.type is logs for log records, metrics for data points, and traces for spans, and is static. ¹ In a special case with mapping::mode: bodymap, data_stream.type field (valid values: logs, metrics) can be dynamically set from attributes. The resulting documents will contain the corresponding data_stream.* fields, see restrictions applied to Data Stream Fields.
   1. data_stream.dataset or data_stream.namespace in attributes (precedence: log record / data point / span attribute > scope attribute > resource attribute)
   2. Otherwise, if a scope attribute with the name encoding.format exists and contains a string value, data_stream.dataset will be set to this value. Note that while enabled by default, this behaviour is considered experimental. Some encoding extensions set this field (e.g. awslogsencodingextension), but it is not yet part of Semantic Conventions. There is the potential that the name of this routing field evolves as the discussion progresses in SemConv.
   3. Otherwise, if scope name matches regex /receiver/(\w*receiver) or /connector/(\w*connector), data_stream.dataset will be capture group #1
   4. Otherwise, data_stream.dataset falls back to generic and data_stream.namespace falls back to default.

• logs_index (optional): The index or data stream name to publish logs (and span events in OTel mapping mode) to. logs_index should be empty unless all logs should be sent to the same index.
• logs_dynamic_index (optional): uses resource, scope, or log record attributes to dynamically construct index name.
  • enabled(DEPRECATED): No-op. Documents are now always routed dynamically unless logs_index is not empty. Will be removed in a future version.
• metrics_index (optional): The index or data stream name to publish metrics to. metrics_index should be empty unless all metrics should be sent to the same index. Note that metrics support is currently in development.
• metrics_dynamic_index (optional): uses resource, scope or data point attributes to dynamically construct index name.
  • enabled(DEPRECATED): No-op. Documents are now always routed dynamically unless metrics_index is not empty. Will be removed in a future version.
• traces_index (optional): The index or data stream name to publish traces to. traces_index should be empty unless all traces should be sent to the same index.
• traces_dynamic_index (optional): uses resource, scope, or span attributes to dynamically construct index name.
  • enabled(DEPRECATED): No-op. Documents are now always routed dynamically unless traces_index is not empty. Will be removed in a future version.
• logstash_format (optional): Logstash format compatibility. Logs, metrics and traces can be written into an index in Logstash format.
  • enabled(default=false): Enable/disable Logstash format compatibility. When logstash_format::enabled is true, the index name is composed using the above dynamic routing rules as prefix and the date as suffix, e.g: If the computed index name is logs-generic-default, the resulting index will be logs-generic-default-YYYY.MM.DD. The last string appended belongs to the date when the data is being generated.
  • prefix_separator(default=-): Set a separator between logstash_prefix and date.
  • date_format(default=%Y.%m.%d): Time format (based on strftime) to generate the second part of the Index name.
• logs_dynamic_id (optional): Dynamically determines the document ID to be used in Elasticsearch based on a log record attribute.
  • enabled(default=false): Enable/Disable dynamic ID for log records. If elasticsearch.document_id exists and is not an empty string in the log record attributes, it will be used as the document ID. Otherwise, the document ID will be generated by Elasticsearch. The attribute elasticsearch.document_id is removed from the final document when the otel mapping mode is used. See Setting a document id dynamically.
• traces_dynamic_id (optional): Dynamically determines the document ID to be used in Elasticsearch based on a span attribute.
  • enabled(default=false): Enable/Disable dynamic ID for spans. If elasticsearch.document_id exists and is not an empty string in the span attributes, it will be used as the document ID. For span events, this only applies when using otel mapping mode (where span events are stored as separate documents). Otherwise, the document ID will be generated by Elasticsearch. The attribute elasticsearch.document_id is removed from the final document when the otel mapping mode is used. See Setting a document id dynamically.

​

Document routing exceptions for OTel data mode

1. “Static mode”: Span events are separate documents routed to logs_index if non-empty.
2. “Dynamic - Index attribute mode”: Span events are separate documents routed using attribute elasticsearch.index (precedence: span event attribute > scope attribute > resource attribute) if the attribute exists.
3. “Dynamic - Data stream routing mode”:

• For all documents, data_stream.dataset will always be appended with .otel.
• A special case to (3)(1) in Elasticsearch document routing, span events are separate documents that have data_stream.type: logs and are routed using data stream attributes (precedence: span event attribute > scope attribute > resource attribute)

​

Elasticsearch document mapping

• mapping:
  • mode (DEPRECATED): The mapping mode if supplied via config file is ignored. Use the X-Elastic-Mapping-Mode client metadata key or the elastic.mapping.mode scope attribute instead. If not specified via these methods, the default mapping mode is otel.
  • allowed_modes (defaults to all mapping modes): A list of allowed mapping modes.

• none
• ecs
• otel
• raw
• bodymap

​

Migration: Setting mapping mode via scope attribute

processors:
  transform:
    log_statements:
      - context: scope
        statements:
          - set(attributes["elastic.mapping.mode"], "otel")
    trace_statements:
      - context: scope
        statements:
          - set(attributes["elastic.mapping.mode"], "otel")
    metric_statements:
      - context: scope
        statements:
          - set(attributes["elastic.mapping.mode"], "otel")
exporters:
  elasticsearch:
    endpoint: https://elasticsearch:9200
service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [transform]
      exporters: [elasticsearch]

[!NOTE] The scope attribute elastic.mapping.mode takes precedence over the X-Elastic-Mapping-Mode client metadata. The attribute will be excluded from the final document sent to Elasticsearch.

[!NOTE] otel and ecs mapping modes require Elasticsearch 8.12 or above². otel mode works best with Elasticsearch 8.16 or above³.

​

OTel mapping mode

​

ECS mapping mode

[!WARNING] The ECS mode mapping mode is currently undergoing changes, and its behaviour is unstable.

​

Bodymap mapping mode

[!WARNING] The Bodymap mode mapping mode is currently undergoing changes, and its behaviour is unstable.

​

Default (none) mapping mode

​

Raw mapping mode

• In none mode attributes are mapped with an Attributes. prefix, while in raw mode they are not.
• In none mode span events are mapped with an Events. prefix, while in raw mode they are not.

​

Elasticsearch ingest pipeline

• pipeline (optional): ID of an Elasticsearch Ingest pipeline used for processing documents published by the exporter.
• logs_dynamic_pipeline (optional): Dynamically determines the ingest pipeline to be used in Elasticsearch based on attributes in the log signal.
  • enabled(default=false): Enable/Disable dynamic pipeline. If elasticsearch.ingest_pipeline attribute exists in the log record attributes and is not an empty string, it will be used as the Elasticsearch ingest pipeline. This currently only applies to the log signal. The attribute elasticsearch.ingest_pipeline is removed from the final document when the otel mapping mode is used.

​

Elasticsearch bulk indexing

• num_workers (DEPRECATED, use sending_queue::num_consumers instead): This config is deprecated and will be used to configure sending_queue::num_consumers if sending_queue::num_consumers is not explicitly defined. Number of workers publishing bulk requests concurrently.
• flush (DEPRECATED, use sending_queue instead): This config is deprecated and will be used to configure different options for sending_queue if sending_queue options are not explicitly defined. Event bulk indexer buffer flush settings
  • bytes (DEPRECATED, use sending_queue::batch::max_size instead): This config is deprecated and will be used to configure sending_queue::batch::max_size if sending_queue::batch::max_size is not explicitly defined. See the sending_queue::batch::max_size for more details.
  • interval (DEPRECATED, use sending_queue::batch::flush_timeout instead): This config is deprecated and will be used to configure sending_queue::batch::flush_timeout if sending_queue::batch::flush_timeout is not explicitly defined. See the sending_queue::batch::flush_timeout for more details.
• retry: Elasticsearch bulk request retry settings
  • enabled (default=true): Enable/Disable request retry on error. Failed requests are retried with exponential backoff.
  • max_requests (DEPRECATED, use retry::max_retries instead): Number of HTTP request retries including the initial attempt. If used, retry::max_retries will be set to max_requests - 1.
  • max_retries (default=2): Number of HTTP request retries. To disable retries, set retry::enabled to false instead of setting max_retries to 0.
  • initial_interval (default=100ms): Initial waiting time if a HTTP request failed.
  • max_interval (default=1m): Max waiting time if a HTTP request failed.
  • retry_on_status (default=[429]): Status codes that trigger request or document level retries. Request level retry and document level retry status codes are shared and cannot be configured separately. To avoid duplicates, it defaults to [429].
• sending_queue: Configures the queueing and batching behaviour. Below are the defaults (which may vary from standard defaults), for full configuration check the exporterhelper docs.
  • enabled (default=true): Enable queueing and batching behaviour.
  • num_consumers (default=10): Number of consumers that dequeue batches.
  • wait_for_result (default=false): If true, blocks incoming requests until processed.
  • block_on_overflow (default=false): If true, blocks the request until the queue has space.
  • sizer (default=requests): Measure queueing by requests.
  • queue_size (default=10): Maximum size the queue can accept.
  • batch:
    • flush_timeout (default=10s): Time after which batch is exported irrespective of other settings.
    • sizer (default=bytes): Size batches by bytes. Note that bytes here are based on the pdata model and not on the NDJSON docs that will constitute the bulk indexer requests. To address this discrepancy, the bulk indexers could also flush when their size exceeds the configured max_size due to size of pdata model being smaller than their corresponding NDJSON encoding.
    • min_size (default=1MB): Min size of the batch.
    • max_size (default=5MB): Max size of the batch. This value should be much lower than Elasticsearch’s http.max_content_length config to avoid HTTP 413 Entity Too Large error. It is recommended to keep this value under 5MB.

​

Bulk indexing error response

• include_source_on_error:
  • true: Enables bulk index responses to include source document on error. Requires Elasticsearch 8.18+. WARNING: the exporter may log error responses containing request payload, causing potential sensitive data to be exposed in logs.
  • false: Disables including source document on bulk index error responses. Requires Elasticsearch 8.18+.
  • null (default): Backward-compatible option for older Elasticsearch versions. By default, the error reason is discarded from bulk index responses entirely, i.e. only error type is returned.

​

Elasticsearch node discovery

• discover:
  • on_start (optional): If enabled the exporter queries Elasticsearch for all known nodes in the cluster on startup.
  • interval (optional): Interval to update the list of Elasticsearch nodes.

​

Telemetry settings

• telemetry:
  • log_request_body (default=false): Logs Elasticsearch client request body as a field in a log line at DEBUG level. It requires service::telemetry::logs::level to be set to debug. WARNING: Enabling this config may expose sensitive data.
  • log_response_body (default=false): Logs Elasticsearch client response body as a field in a log line at DEBUG level. It requires service::telemetry::logs::level to be set to debug. WARNING: Enabling this config may expose sensitive data.
  • log_failed_docs_input (default=false): Include the input (action line and document line) causing indexing error under input field in a log line at DEBUG level. It requires service::telemetry::logs::level to be set to debug. WARNING: Enabling this config may expose sensitive data.
  • log_failed_docs_input_rate_limit (default=“1s”): Rate limiting of logs emitted by log_failed_docs_input config, e.g. “1s” means roughly 1 log line per second. A zero or negative value disables rate limiting.

​

Metadata keys

• metadata_keys (optional): List of metadata keys that will be used to partition the data into batches if sending_queue is enabled with batching support. With batching enabled only these metadata keys are guaranteed to be propagated. The keys will also be used to enrich the exporter’s internal telemetry if defined. The keys are extracted from the client metadata available via the context and added to the internal telemetry as attributes.

​

Exporting metrics

• Gauge
• Sum
• Histogram (Delta temporality only)
• Exponential histogram (Delta temporality only)
• Summary

​

Metrics dynamic templates

A map from the full name of fields to the name of dynamic templates. It defaults to an empty map. If a name matches a dynamic template, that template will be applied regardless of other match predicates defined in the template. If a field is already defined in the mapping, then this parameter won’t be used.

• OTel: Each metric is written under the metrics object; the bulk action maps full field names (e.g. metrics.my_metric) to one of the OTel template names above based on metric type (histogram, summary, gauge, or counter) and value type.
• ECS: Each metric is written as a top-level field metric.<name>; the bulk action maps that field name to one of the ECS/APM template names (histogram_metrics, summary_metrics, or double_metrics for gauges and counters).

​

Exporting profiles

exporters:
  elasticsearch:
    endpoint: https://elastic.example.com:9200
    mapping:
      mode: otel

[!IMPORTANT] For the Elasticsearch Exporter to be able to export Profiles data, Universal Profiling needs to be installed in the database. See the Universal Profiling getting started documentation You will need to use the Elasticsearch endpoint, with an Elasticsearch API key.

​

ECS Mapping

​

Resource attribute mapping

​

Log record attribute mapping

​

Span attribute mapping

​

Compound Mapping

​

host.name and host.hostname

​

@timestamp

​

Setting a document id dynamically

processors:
  transform/es-doc-id:
    error_mode: ignore
    log_statements:
      - context: log
        condition: attributes["event_name"] != null && attributes["event_creation_time"] != null
        statements:
          - set(attributes["elasticsearch.document_id"], Concat(["log", attributes["event_name"], attributes["event_creation_time"], "-"))

exporters:
  elasticsearch:
    mapping:
      mode: otel  # Required for span events to be separate documents
    traces_dynamic_id:
      enabled: true

processors:
  transform/es-doc-id-traces:
    error_mode: ignore
    trace_statements:
      - context: span
        statements:
          # Set ID for spans
          - set(attributes["elasticsearch.document_id"], Concat([trace_id.string, span_id.string], "-"))
      - context: spanevent
        statements:
          # Set ID for span events (only works in otel mapping mode)
          - set(attributes["elasticsearch.document_id"], Concat([trace_id.string, span_id.string, name], "-"))

​

Known issues

​

version_conflict_engine_exception

• minimum Elasticsearch Exporter version 0.121.0
• minimum Elasticsearch version 8.17.6, 8.18.1, 8.19.0, 9.0.1, or 9.1.0

PUT _component_template/metrics-otel@custom
{
  "template": {
    "mappings": {
      "properties": {
        "_metric_names_hash": {
          "type": "keyword",
          "time_series_dimension": true
        }
      }
    }
  }
}

1. If the data is not sent in otel mapping mode to metrics-*.otel-* data streams, the metrics name fingerprint is not applied. This can happen for OTel host and k8s metrics that the elasticinframetricsprocessor has translated to the format the host and k8s dashboards in Kibana can consume. If these metrics arrive in the elasticsearchexporter in different batches, they will not be grouped to the same document. This can cause the version_conflict_engine_exception error. Try to remove the batchprocessor from the pipeline (or set send_batch_max_size: 0) to ensure metrics are not split into different batches. This gives the exporter the opportunity to group all related metrics into the same document.
2. Otherwise, check your metrics pipeline setup for misconfiguration that causes an actual violation of the single writer principle. This means that the same metric with the same dimensions is sent from multiple sources, which is not allowed in the OTel metrics data model.

​

flush failed (400) illegal_argument_exception

error   [email protected]/bulkindexer.go:343       bulk indexer flush error
{
  "otelcol.component.id": "elasticsearch",
  "otelcol.component.kind": "Exporter",
  "otelcol.signal": "logs",
  "error": "flush failed (400): {\"error\":{\"type\":\"illegal_argument_exception\",\"caused_by\":{}}}"
}

​

”dropping cumulative temporality histogram” and “dropping cumulative temporality exponential histogram”

warn    [email protected]/exporter.go:340  validation errors
{
  "resource": {
    "service.instance.id": "33ffe7e8-e944-4f92-8fce-9094f4b61d1d",
    "service.name": "./elastic-agent",
    "service.version": "9.1.5"
  },
  "otelcol.component.id": "elasticsearch/otel",
  "otelcol.component.kind": "exporter",
  "otelcol.signal": "metrics",
  "error": "dropping cumulative temporality histogram \"http.client.request.duration\""
}

• Export histogram metrics using delta temporality, or
• Apply a cumulativetodelta processor. For more details, see Metrics data ingestion.

​

Attributes

​

Configuration

​

Example Configuration

elasticsearch:
  endpoints: [https://elastic.example.com:9200]
elasticsearch/trace:
  tls:
    insecure: false
  endpoints: [https://elastic.example.com:9200]
  timeout: 2m
  headers:
    myheader: test
  traces_index: trace_index
  traces_dynamic_index:
    enabled: false
  logs_dynamic_index:
    enabled: false
  metrics_dynamic_index:
    enabled: false
  pipeline: mypipeline
  user: elastic
  password: search
  api_key: AvFsEiPs==
  discover:
    on_start: true
  retry:
    max_retries: 5
    retry_on_status:
      - 429
      - 500
elasticsearch/metric:
  tls:
    insecure: false
  endpoints: [http://localhost:9200]
  metrics_index: my_metric_index
  traces_dynamic_index:
    enabled: false
  logs_dynamic_index:
    enabled: false
  metrics_dynamic_index:
    enabled: false
  timeout: 2m
  headers:
    myheader: test
  pipeline: mypipeline
  user: elastic
  password: search
  api_key: AvFsEiPs==
  discover:
    on_start: true
  retry:
    max_retries: 5
    retry_on_status:
      - 429
      - 500
elasticsearch/log:
  tls:
    insecure: false
  endpoints: [http://localhost:9200]
  logs_index: my_log_index
  traces_dynamic_index:
    enabled: false
  logs_dynamic_index:
    enabled: false
  metrics_dynamic_index:
    enabled: false
  timeout: 2m
  headers:
    myheader: test
  pipeline: mypipeline
  user: elastic
  password: search
  api_key: AvFsEiPs==
  discover:
    on_start: true
  retry:
    max_retries: 5
    retry_on_status:
      - 429
      - 500
elasticsearch/logstash_format:
  endpoints: [http://localhost:9200]
  logstash_format:
    enabled: true
elasticsearch/raw:
  endpoints: [http://localhost:9200]
  mapping:
    mode: raw
elasticsearch/cloudid:
  cloudid: foo:YmFyLmNsb3VkLmVzLmlvJGFiYzEyMyRkZWY0NTY=
elasticsearch/confighttp_endpoint:
  endpoint: https://elastic.example.com:9200
elasticsearch/compression_none:
  endpoint: https://elastic.example.com:9200
  compression: none
elasticsearch/compression_gzip:
  endpoint: https://elastic.example.com:9200
  compression: gzip
elasticsearch/include_source_on_error:
  endpoint: https://elastic.example.com:9200
  include_source_on_error: true
elasticsearch/metadata_keys:
  endpoint: https://elastic.example.com:9200
  metadata_keys:
    - x-test-1
    - x-test-2
elasticsearch/sendingqueue_disabled:
  endpoint: https://elastic.example.com:9200
  sending_queue:
    enabled: false
elasticsearch/sendingqueue_enabled:
  endpoint: https://elastic.example.com:9200
  sending_queue:
    enabled: true
    sizer: requests
    num_consumers: 100
    batch:
      flush_timeout: 1s
      sizer: items
      min_size: 1000
      max_size: 5000
elasticsearch/backward_compat_for_deprecated_cfgs/new_config_takes_priority:
  endpoint: https://elastic.example.com:9200
  # Should be ignored and left as-is
  num_workers: 11
  flush:
    interval: 11s
    bytes: 1001
  # Should take precedence
  sending_queue:
    enabled: true
    sizer: requests
    num_consumers: 111
    batch:
      flush_timeout: 111s
      max_size: 1_000_001
      sizer: bytes
elasticsearch/backward_compat_for_deprecated_cfgs/fallback_to_old_cfg:
  endpoint: https://elastic.example.com:9200
  # Should be used to set sending_queue config
  num_workers: 11
  flush:
    interval: 11s
    bytes: 1_000_001
  sending_queue:
    enabled: true
    sizer: requests
    batch:
      sizer: bytes
elasticsearch/suppress_conflict_errors:
  endpoint: https://elastic.example.com:9200
  suppress_conflict_errors: true