Awss3 Receiver

Available in: contrib Maintainers: @atoulme, @adcharre Source: opentelemetry-collector-contrib

Supported Telemetry

Overview

Receiver for retrieving telemetry data (traces, metrics, and logs) previously stored in S3. This receiver is commonly used with the AWS S3 Exporter but can process any S3 objects containing telemetry data in supported formats. The receiver supports:

Traces, Metrics, and Logs - All three OpenTelemetry signal types
Multiple formats - OTLP (JSON and Protocol Buffers), plus custom formats via encoding extensions
Two retrieval modes:
- Time-based: Fetch data from a specific time range using S3 key partitioning
- Event-driven: Process new objects as they arrive via SQS notifications

Configuration

The following exporter configuration parameters are supported.

Name	Description	Default	Required
`starttime`	The time at which to start retrieving data.		Required if fetching by time
`endtime`	The time at which to stop retrieving data.		Required if fetching by time
`s3downloader:`
`region`	AWS region.	”us-east-1”	Optional
`s3_bucket`	S3 bucket		Required
`s3_prefix`	prefix for the S3 key (root directory inside bucket).		Required
`s3_partition_format`	Format for the partition key, See strftime for format specification.	`"year=%Y/month=%m/day=%d/hour=%H/minute=%M"`	Optional
`s3_partition_timezone`	IANA timezone name applied when formatting the partition key.	Local time	Optional
`file_prefix`	file prefix defined by user		Optional
`file_prefix_include_telemetry_type`	whether to append `<telemetry_type>_` to the file prefix when building S3 keys	true	Optional
`endpoint`	overrides the endpoint used by the exporter instead of constructing it from `region` and `s3_bucket`		Optional
`endpoint_partition_id`	partition id to use if `endpoint` is specified.	”aws”	Optional
`s3_force_path_style`	set this to `true` to force the request to use path-style addressing	false	Optional
`sqs:`
`queue_url`	The URL of the SQS queue that receives S3 bucket notifications		Required if fetching by SQS notification
`region`	AWS region of the SQS queue		Required if fetching by SQS notification
`endpoint`	Custom endpoint for the SQS service		Optional
`max_number_of_messages`	Maximum number of messages to retrieve in a single SQS request	10	Optional
`wait_time_seconds`	Wait time in seconds for long polling SQS requests	20	Optional
`encodings:`	An array of entries with the following properties:		Optional
`extension`	Extension to use for decoding a key with a matching suffix.		Required
`suffix`	Key suffix to match against.		Required
`notifications:`
`opampextension`	Name of the OpAMP Extension to use to send ingest progress notifications.
`tag_object_after_ingestion`	If enabled the receiver will attempt to tag the object after successfully ingesting it.	false	Optional
`skip_ingesting_tagged_objects`	If enabled the receiver will skip objects tagged by `tag_object_after_ingestion`. This can be used as a checkpointing mechanism, and requires an additional `s3:GetObjectTagging` permission	false	Optional

There are two modes of operation:

Time Range Mode - Specify starttime and endtime to fetch data from a specific time range.
SQS Message Mode - Subscribe to SQS messages to process new objects as they arrive.

SQS Message Configuration

The receiver can subscribe to an SQS queue that receives S3 event notifications:

sqs:
  # Required: The ARN of the SQS queue that receives S3 bucket notifications
  queue_url: "https:https://sqs.us-east-1.amazonaws.com/123456789012/test-queue"
  # Required: The AWS region of the SQS queue
  region: "us-east-1"

Note: You must configure your S3 bucket to send event notifications to the SQS queue. Time-based configuration (starttime/endtime) and SQS configuration cannot be used together.

Time format for `starttime` and `endtime`

The starttime and endtime fields are used to specify the time range for which to retrieve data. The time format is either RFC3339,YYYY-MM-DD HH:MM or simply YYYY-MM-DD, in which case the time is assumed to be 00:00.

Encodings

By default, the receiver understands the following encodings:

otlp_json (OpenTelemetry Protocol format represented as json) with a suffix of .json
otlp_proto (OpenTelemetry Protocol format represented as Protocol Buffers) with a suffix of .binpb

The encodings options allows you to specify Encoding Extensions to use to decode keys with matching suffixes.

Example Configuration

extension:
  # example of text encoding extension
  text_encoding:
    encoding: utf8
    marshaling_separator: "\n"
    unmarshaling_separator: "\r?\n"

receivers:
  awss3:
    starttime: "2024-01-01 01:00"
    endtime: "2024-01-02"
    s3downloader:
        region: "us-west-1"
        s3_bucket: "mybucket"
        s3_prefix: "trace"
        s3_partition_format: "year=%Y/month=%m/day=%d/hour=%H/minute=%M"
        s3_partition_timezone: "UTC"
        file_prefix_include_telemetry_type: true
    encodings:
      - extension: text_encoding
        suffix: ".txt"

receivers:
  awss3/sqs_traces:
    s3downloader:
      region: us-east-1
      s3_bucket: mybucket
      s3_prefix: mytrace
    sqs:
      queue_url: "https://sqs.us-east-1.amazonaws.com/123456789012/test-queue"
      region: "us-east-1"

exporters:
  otlp_grpc:
    endpoint: otelcol:4317

service:
  pipelines:
    traces:
      receivers: [awss3/traces]
      exporters: [otlp_grpc]

    traces/sqs:
      receivers: [awss3/sqs_traces]
      exporters: [otlp_grpc]

Notifications

The receiver can send notifications of ingest progress to an OpAmp server using the custom message capability of “org.opentelemetry.collector.receiver.awss3” and message type “TimeBasedIngestStatus”. The format of the notifications is a ProtoBuf formatted OLTP logs message with a single Log Record. The body of the record is set to status and the timestamp of the record is used to hold the ingest time. The record also has the following attributes:

Attribute	Description
`telemetry_type`	The type of telemetry being ingested. One of “traces”, “metrics”, or “logs”.
`ingest_status`	The status of the data ingestion. One of “ingesting”, “failed”, or “completed”.
`start_time`	The time to start retrieving data as an Int64, nanoseconds since Unix epoch.
`end_time`	The time to stop retrieving data as an Int64, nanoseconds since Unix epoch.
`failure_message`	Error message if `ingest_status` is “failed”.

The “ingesting” status is sent at the beginning of the ingest process before data has been retrieved for the specified time. If during the processing of the data an error occurs a status message with ingest_status set to “failed” status with the time of the data being ingested when the failure occurred. If the ingest process completes successfully a status message with ingest_status set to “completed” is sent.

Object Lifecycle Management

If the tag_object_after_ingestion is enabled the receiver will make a best-effort attempt to tag objects with otel-collector:status = ingested after they are processed by the pipeline. This requires an additional s3:PutObjectTagging permission. This tag can then be used with a lifecycle policy to expire ingested objects or transition them to cheaper storage classes.

Configuration

Example Configuration

awss3:
awss3/1:
  s3downloader:
    s3_bucket: abucket
    s3_partition_format: "%Y/%m"
  starttime: "a date"
  endtime: "2024-02-03a"
awss3/2:
  s3downloader:
    s3_bucket: abucket
  starttime: "2024-01-31 15:00"
  endtime: "2024-02-03"
awss3/3:
  s3downloader:
    s3_bucket: abucket
    s3_partition_format: "year=%Y/month=%m/day=%d/hour=%H"
    s3_partition_timezone: "Asia/Tokyo"
    file_prefix: "otel"
    file_prefix_include_telemetry_type: false
    tag_object_after_ingestion: false
    skip_ingesting_tagged_objects: false
  starttime: "2024-01-31 15:00"
  endtime: "2024-02-03"
  encodings:
    - extension: "foo/bar"
      suffix: "baz"
    - extension: "nop/nop"
      suffix: "nop"
  notifications:
    opampextension: "opamp/bar"
awss3/4:
  s3downloader:
    s3_bucket: abucket
    tag_object_after_ingestion: true
    skip_ingesting_tagged_objects: true
  starttime: "2024-01-31T15:00:00Z"
  endtime: "2024-02-03T00:00:00Z"
awss3/5:
  s3downloader:
    s3_bucket: abucket
  sqs:
    queue_url: "https://sqs.us-east-1.amazonaws.com/123456789012/test-queue"
    region: "us-east-1"
    endpoint: "http://localhost:4575"

Last generated: 2026-08-24

​Awss3 Receiver

​Supported Telemetry

​Overview

​Overview

​Configuration

​SQS Message Configuration

​Time format for starttime and endtime

​Encodings

​Example Configuration

​Notifications

​Object Lifecycle Management

​Configuration

​Example Configuration

Awss3 Receiver

Supported Telemetry

Overview

Overview

Configuration

SQS Message Configuration

Time format for `starttime` and `endtime`

Encodings

Example Configuration

Notifications

Object Lifecycle Management

Configuration

Example Configuration