Googlecloudstorage Exporter

Available in: contrib Maintainers: @constanca-m, @braydonk Source: opentelemetry-collector-contrib

Supported Telemetry

Overview

This exporter writes received OpenTelemetry data to a cloud storage bucket.

Configuration

Name	Description	Required	Default
`encoding`	The encoding extension ID to use for marshaling logs and traces. If left empty, `plog.JSONMarshaler` will be used for logs and `ptrace.JSONMarshaler` will be used for traces.	No
`bucket.project_id`	The project where the bucket will be created or where it exists. If left empty, it will query the metadata endpoint. It requires the collector to be running in a Google Cloud environment.	No
`bucket.name`	Name for the bucket storage.	Yes
`bucket.file_prefix`	Prefix for the created filename. This prefix is applied after the partition path (if any).	No	`logs`
`bucket.partition`	Configuration for time-based partitioning. See below for details.	No
`bucket.reuse_if_exists`	Controls bucket creation behavior. If `true`, checks if bucket exists and uses it (requires `storage.buckets.get` permission on the bucket); fails if bucket doesn’t exist. If `false`, attempts to create bucket; fails if bucket already exists (requires `storage.buckets.create` permission at project level). Set to `true` when the service account lacks project-level bucket creation permissions but has bucket-level permissions.	No	`false`
`bucket.region`	Region where the bucket will be created or where it exists. If left empty, it will query the metadata endpoint. It requires the collector to be running in a Google Cloud environment.	Yes
`bucket.compression`	Compression algorithm used to compress data before uploading. Valid values are `gzip`, `zstd`, or no value set for no compression.	No
`resource_attrs_to_gcs`	Mapping of GCS upload configuration values to resource attribute values. See resource_attrs_to_gcs below.	No
`timeout`	Time to wait per individual attempt to send data to the backend. See `exporterhelper` docs for details.	No	`5s`
`retry_on_failure`	Configuration for how to retry failed requests. See `exporterhelper` docs for details.	No
`sending_queue`	Configuration for the internal sending queue. See `exporterhelper` docs for details.	No

Partition Configuration

The bucket.partition configuration allows you to organize files into time-based folders.

Name	Description	Required	Default
`format`	Time format string for time-based partitions. If set, formatted UTC time is prepended to the filename. Supports strftime format (e.g., `year=%Y/month=%m`).	No
`prefix`	Prefix for the partition folder structure. If set, this value is prepended to the partition path.	No

resource_attrs_to_gcs

Name	Description	Required	Default
`prefix`	Resource attribute whose value is inserted as a partition path segment between `bucket.partition.prefix` and the time-based `bucket.partition.format`. If the attribute is present on the data, its value is used; otherwise the segment is omitted.	No

When resource_attrs_to_gcs.prefix is configured, a partition segment is derived from the specified resource attribute, allowing data to be routed into per-attribute folders (for example, one folder per host or service). The resulting layout is <partition.prefix>/<attribute value>/<partition.format>/..., so the existing bucket.partition.prefix is preserved and the attribute segment is appended after it. The attribute value is stringified, so non-string attributes are converted to their string form (for example, the int 42 becomes 42 and a bool becomes true); complex types such as maps or slices render as their textual representation and are not recommended. Because resource attribute values are arbitrary and may be untrusted, the value is normalized with the standard library path.Clean before being used as an object-name segment: duplicate slashes are collapsed and empty, . and .. segments are removed (so the value cannot traverse above its segment root). Leading and trailing / are trimmed, while internal / are preserved so a value can span multiple folder levels (e.g. service-a/logs). If the value normalizes to an empty string, no extra segment is added. Note that high-cardinality attribute values produce a correspondingly large number of object key prefixes.

Note: the attribute value is read from the first resource of each exported batch. If a batch can contain multiple distinct attribute values and you need strict per-value routing, separate the data upstream first using the routingconnector or groupbyattrsprocessor.

Example

Here is an example configuration for this exporter:

exporters:
  google_cloud_storage:
    encoding: text_encoding
    bucket:
      name: bucket-test
      project_id: project-test
      reuse_if_exists: true
      region: europe-west1
      file_prefix: app-logs
      partition:
        prefix: archive
        format: "year=%Y/month=%m/day=%d/hour=%H"

extensions:
  # text encoding to ensure only the body is placed in the bucket
  text_encoding:

Routing by Resource Attribute

This configuration writes each service’s data into its own folder by deriving a partition segment from the service.name resource attribute, while keeping a static storage prefix:

exporters:
  google_cloud_storage:
    bucket:
      name: archive-bucket
      region: europe-west1
      file_prefix: logs
      partition:
        prefix: storage
        format: "%Y-%m-%d/%H"
    resource_attrs_to_gcs:
      prefix: service.name

Given a resource with service.name = serviceA, an object is written to:

storage/serviceA/2026-06-17/14/logs_<uuid>

If the service.name attribute is absent, the attribute segment is omitted and the object is written to storage/2026-06-17/14/logs_<uuid>.

Resiliency Settings

This exporter supports standard OpenTelemetry Collector resiliency settings. You can configure timeouts, retry behaviors, and a sending queue to ensure reliable data delivery to Google Cloud Storage.

exporters:
  google_cloud_storage:
    bucket:
      name: bucket-test
      region: europe-west1
    timeout: 5s
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 1000

For more details on how to configure these specific sections, refer to the exporterhelper configuration documentation.

Retry Behavior

This exporter utilizes two independent layers of retries: Google Cloud SDK Retries (always on): The underlying Google Cloud Go SDK automatically retries transient errors (like network blips) in the background. By default, the SDK handles its own exponential backoff and idempotency checks. Exporter Helper Retries (optional via retry_on_failure): This is the OpenTelemetry-level retry mechanism. It integrates with the sending queue to ensure data is not dropped during prolonged outages. Important Notes on Retries: If the Google Cloud SDK determines an error is permanent (e.g., unauthorized access) or not idempotent, the exporter will immediately wrap it as a permanent error. This halts the retry_on_failure loop and drops the payload to prevent infinite queue blocking. You can find more information on what the SDK considers retryable in the Google Cloud Storage Retry Strategy documentation. Because file names are generated dynamically using a UUID and timestamp, each retry attempt by the OpenTelemetry retry_on_failure mechanism generates a new filename. In rare cases (such as when GCS commits an object but returns a transient error before the client receives acknowledgment) this can result in duplicate blobs in GCS.

Compression Example

exporters:
  google_cloud_storage:
    bucket:
      name: compressed-logs-bucket
      project_id: my-project
      region: us-central1
      compression: gzip
      file_prefix: logs
      partition:
        format: "year=%Y/month=%m/day=%d"

Using with Bucket-Level Permissions Only

When the service account lacks project-level bucket creation permissions but has bucket-level permissions:

exporters:
  google_cloud_storage:
    bucket:
      name: existing-bucket
      project_id: my-project
      region: us-central1
      reuse_if_exists: true
      file_prefix: collector-logs

With reuse_if_exists: true, the exporter checks if the bucket exists using storage.buckets.get permission on that specific bucket (not project-level). If the bucket doesn’t exist, the exporter will fail with an error. This allows service accounts with only bucket-level permissions to work without requiring project-level bucket creation permissions.

Configuration

Example Configuration

google_cloud_storage:
  encoding: test
  bucket:
    region: test-region
    name: test-bucket
    project_id: test-project-id

google_cloud_storage/with_partition:
  encoding: test
  bucket:
    region: test-region
    name: test-bucket
    project_id: test-project-id
    partition:
      format: year=%Y
      prefix: my-logs

google_cloud_storage/with_resource_attrs_to_gcs:
  encoding: test
  bucket:
    region: test-region
    name: test-bucket
    project_id: test-project-id
    partition:
      format: "%Y-%m-%d/%H"
      prefix: storage
  resource_attrs_to_gcs:
    prefix: service.name

google_cloud_storage/with_resiliency:
  encoding: test
  timeout: 3s
  sending_queue:
    num_consumers: 7
    queue_size: 123
  retry_on_failure:
    enabled: true
    initial_interval: 1s
    max_interval: 5s
    max_elapsed_time: 20s
  bucket:
    region: test-region
    name: test-bucket
    project_id: test-project-id

google_cloud_storage/empty_bucket_name:
  encoding: test
  bucket:
    region: test-region
    project_id: test-project-id

google_cloud_storage/invalid_partition_format:
  encoding: test
  bucket:
    region: test-region
    name: test-bucket
    project_id: test-project-id
    partition:
      format: year=%invalid

google_cloud_storage/with_gzip_compression:
  encoding: test
  bucket:
    region: test-region
    name: test-bucket
    project_id: test-project-id
    compression: gzip

google_cloud_storage/with_zstd_compression:
  encoding: test
  bucket:
    region: test-region
    name: test-bucket
    project_id: test-project-id
    compression: zstd

google_cloud_storage/unsupported_compression:
  encoding: test
  bucket:
    region: test-region
    name: test-bucket
    project_id: test-project-id
    compression: snappy

google_cloud_storage/with_universe_domain:
  encoding: test
  universe_domain: apis.example.com
  bucket:
    region: test-region
    name: test-bucket
    project_id: test-project-id

Last generated: 2026-08-24

​Googlecloudstorage Exporter

​Supported Telemetry

​Overview

​Configuration

​Partition Configuration

​resource_attrs_to_gcs

​Example

​Routing by Resource Attribute

​Resiliency Settings

​Retry Behavior

​Compression Example

​Using with Bucket-Level Permissions Only

​Configuration

​Example Configuration

Googlecloudstorage Exporter

Supported Telemetry

Overview

Configuration

Partition Configuration

resource_attrs_to_gcs

Example

Routing by Resource Attribute

Resiliency Settings

Retry Behavior

Compression Example

Using with Bucket-Level Permissions Only

Configuration

Example Configuration