Skip to main content

​
Awss3 Receiver

Status Available in: contrib Maintainers: @atoulme, @adcharre Source: opentelemetry-collector-contrib

​
Supported Telemetry

Logs Metrics Traces

​
Overview

​
Overview

Receiver for retrieving telemetry data (traces, metrics, and logs) previously stored in S3. This receiver is commonly used with the AWS S3 Exporter but can process any S3 objects containing telemetry data in supported formats. The receiver supports:
  • Traces, Metrics, and Logs - All three OpenTelemetry signal types
  • Multiple formats - OTLP (JSON and Protocol Buffers), plus custom formats via encoding extensions
  • Two retrieval modes:
    • Time-based: Fetch data from a specific time range using S3 key partitioning
    • Event-driven: Process new objects as they arrive via SQS notifications

​
Configuration

The following exporter configuration parameters are supported.
NameDescriptionDefaultRequired
starttimeThe time at which to start retrieving data.Required if fetching by time
endtimeThe time at which to stop retrieving data.Required if fetching by time
s3downloader:
regionAWS region.”us-east-1”Optional
s3_bucketS3 bucketRequired
s3_prefixprefix for the S3 key (root directory inside bucket).Required
s3_partition_formatFormat for the partition key, See strftime for format specification."year=%Y/month=%m/day=%d/hour=%H/minute=%M"Optional
s3_partition_timezoneIANA timezone name applied when formatting the partition key.Local timeOptional
file_prefixfile prefix defined by userOptional
file_prefix_include_telemetry_typewhether to append <telemetry_type>_ to the file prefix when building S3 keystrueOptional
endpointoverrides the endpoint used by the exporter instead of constructing it from region and s3_bucketOptional
endpoint_partition_idpartition id to use if endpoint is specified.”aws”Optional
s3_force_path_styleset this to true to force the request to use path-style addressingfalseOptional
sqs:
queue_urlThe URL of the SQS queue that receives S3 bucket notificationsRequired if fetching by SQS notification
regionAWS region of the SQS queueRequired if fetching by SQS notification
endpointCustom endpoint for the SQS serviceOptional
max_number_of_messagesMaximum number of messages to retrieve in a single SQS request10Optional
wait_time_secondsWait time in seconds for long polling SQS requests20Optional
encodings:An array of entries with the following properties:Optional
extensionExtension to use for decoding a key with a matching suffix.Required
suffixKey suffix to match against.Required
notifications:
opampextensionName of the OpAMP Extension to use to send ingest progress notifications.
tag_object_after_ingestionIf enabled the receiver will attempt to tag the object after successfully ingesting it.falseOptional
There are two modes of operation:
  1. Time Range Mode - Specify starttime and endtime to fetch data from a specific time range.
  2. SQS Message Mode - Subscribe to SQS messages to process new objects as they arrive.

​
SQS Message Configuration

The receiver can subscribe to an SQS queue that receives S3 event notifications: 
sqs:
  # Required: The ARN of the SQS queue that receives S3 bucket notifications
  queue_url: "https:https://sqs.us-east-1.amazonaws.com/123456789012/test-queue"
  # Required: The AWS region of the SQS queue
  region: "us-east-1"
Note: You must configure your S3 bucket to send event notifications to the SQS queue. Time-based configuration (starttime/endtime) and SQS configuration cannot be used together.

​
Time format for starttime and endtime

The starttime and endtime fields are used to specify the time range for which to retrieve data. The time format is either RFC3339,YYYY-MM-DD HH:MM or simply YYYY-MM-DD, in which case the time is assumed to be 00:00.

​
Encodings

By default, the receiver understands the following encodings:
  • otlp_json (OpenTelemetry Protocol format represented as json) with a suffix of .json
  • otlp_proto (OpenTelemetry Protocol format represented as Protocol Buffers) with a suffix of .binpb
The encodings options allows you to specify Encoding Extensions to use to decode keys with matching suffixes.

​
Example Configuration

extension:
  # example of text encoding extension
  text_encoding:
    encoding: utf8
    marshaling_separator: "\n"
    unmarshaling_separator: "\r?\n"

receivers:
  awss3:
    starttime: "2024-01-01 01:00"
    endtime: "2024-01-02"
    s3downloader:
        region: "us-west-1"
        s3_bucket: "mybucket"
        s3_prefix: "trace"
        s3_partition_format: "year=%Y/month=%m/day=%d/hour=%H/minute=%M"
        s3_partition_timezone: "UTC"
        file_prefix_include_telemetry_type: true
    encodings:
      - extension: text_encoding
        suffix: ".txt"

receivers:
  awss3/sqs_traces:
    s3downloader:
      region: us-east-1
      s3_bucket: mybucket
      s3_prefix: mytrace
    sqs:
      queue_url: "https://sqs.us-east-1.amazonaws.com/123456789012/test-queue"
      region: "us-east-1"

exporters:
  otlp_grpc:
    endpoint: otelcol:4317

service:
  pipelines:
    traces:
      receivers: [awss3/traces]
      exporters: [otlp_grpc]

    traces/sqs:
      receivers: [awss3/sqs_traces]
      exporters: [otlp_grpc]

​
Notifications

The receiver can send notifications of ingest progress to an OpAmp server using the custom message capability of “org.opentelemetry.collector.receiver.awss3” and message type “TimeBasedIngestStatus”. The format of the notifications is a ProtoBuf formatted OLTP logs message with a single Log Record. The body of the record is set to status and the timestamp of the record is used to hold the ingest time. The record also has the following attributes:
AttributeDescription
telemetry_typeThe type of telemetry being ingested. One of “traces”, “metrics”, or “logs”.
ingest_statusThe status of the data ingestion. One of “ingesting”, “failed”, or “completed”.
start_timeThe time to start retrieving data as an Int64, nanoseconds since Unix epoch.
end_timeThe time to stop retrieving data as an Int64, nanoseconds since Unix epoch.
failure_messageError message if ingest_status is “failed”.
The “ingesting” status is sent at the beginning of the ingest process before data has been retrieved for the specified time. If during the processing of the data an error occurs a status message with ingest_status set to “failed” status with the time of the data being ingested when the failure occurred. If the ingest process completes successfully a status message with ingest_status set to “completed” is sent.

​
Object Lifecycle Management

If the tag_object_after_ingestion is enabled the receiver will make a best-effort attempt to tag objects with otel-collector:status = ingested after they are processed by the pipeline. This requires an additional s3:PutObjectTagging permission. This tag can then be used with a lifecycle policy to expire ingested objects or transition them to cheaper storage classes.

​
Configuration

​
Example Configuration

awss3:
awss3/1:
  s3downloader:
    s3_bucket: abucket
    s3_partition_format: "%Y/%m"
  starttime: "a date"
  endtime: "2024-02-03a"
awss3/2:
  s3downloader:
    s3_bucket: abucket
  starttime: "2024-01-31 15:00"
  endtime: "2024-02-03"
awss3/3:
  s3downloader:
    s3_bucket: abucket
    s3_partition_format: "year=%Y/month=%m/day=%d/hour=%H"
    s3_partition_timezone: "Asia/Tokyo"
    file_prefix: "otel"
    file_prefix_include_telemetry_type: false
    tag_object_after_ingestion: false
  starttime: "2024-01-31 15:00"
  endtime: "2024-02-03"
  encodings:
    - extension: "foo/bar"
      suffix: "baz"
    - extension: "nop/nop"
      suffix: "nop"
  notifications:
    opampextension: "opamp/bar"
awss3/4:
  s3downloader:
    s3_bucket: abucket
    tag_object_after_ingestion: true
  starttime: "2024-01-31T15:00:00Z"
  endtime: "2024-02-03T00:00:00Z"
awss3/5:
  s3downloader:
    s3_bucket: abucket
  sqs:
    queue_url: "https://sqs.us-east-1.amazonaws.com/123456789012/test-queue"
    region: "us-east-1"
    endpoint: "http://localhost:4575"
Last generated: 2026-04-13