> ## Documentation Index
> Fetch the complete documentation index at: https://otel.fyi/llms.txt
> Use this file to discover all available pages before exploring further.
# Googlecloud
> OpenTelemetry exporter for Googlecloud
# Googlecloud Exporter
**Available in:** `contrib`
**Maintainers:** [@aabmass](https://github.com/aabmass), [@dashpole](https://github.com/dashpole), [@braydonk](https://github.com/braydonk), [@jsuereth](https://github.com/jsuereth), [@psx95](https://github.com/psx95), [@ridwanmsharif](https://github.com/ridwanmsharif)
**Source:** [opentelemetry-collector-contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/googlecloudexporter)
## Supported Telemetry
  
## Overview
This exporter can be used to send metrics to [Google Cloud Monitoring](https://cloud.google.com/monitoring)
(formerly Stackdriver), traces to [Google Cloud Trace](https://cloud.google.com/trace),
and logs to [Google Cloud Logging](https://cloud.google.com/logging).
To learn more about instrumentation and observability, including opinionated recommendations
for Google Cloud Observability, visit [Instrumentation and
observability](https://cloud.google.com/stackdriver/docs/instrumentation/overview).
## Getting started
### Prerequisite: Authenticating
In general, authenticating with the Collector exporter follows the same steps as
any other app using the steps documented for [Application Default
Credentials](https://cloud.google.com/docs/authentication/provide-credentials-adc). This
section explains the specific use cases relevant to the exporter.
The exporter relies on GCP client libraries to send data to Google Cloud. Use of these libraries requires the caller (the Collector) to be authenticated with a GCP account and project. This should be done using a [GCP service account](https://cloud.google.com/compute/docs/access/service-accounts) with at minimum the following IAM roles (depending on the type of data you wish to send):
* [Metrics](https://cloud.google.com/iam/docs/understanding-roles#monitoring-roles): `roles/monitoring.metricWriter`
* [Traces](https://cloud.google.com/iam/docs/understanding-roles#cloud-trace-roles): `roles/cloudtrace.agent`
* [Logs](https://cloud.google.com/iam/docs/understanding-roles#logging-roles): `roles/logging.logWriter`
The [Compute Engine default service account](https://cloud.google.com/compute/docs/access/service-accounts#default_service_account) has all of these permissions by default, but if you are running on a different platform or with a different GCP service account you will need to ensure your service account has these permissions.
#### Options for different environments
Depending on the environment where your Collector is running, you can authenticate one of several ways:
**GCE instances**
On GCE it is recommended to use the [GCP service account](https://cloud.google.com/compute/docs/access/service-accounts) associated with your instance. If this is the Compute Engine default service account or another GCP service account with the sufficient IAM permissions, then there is nothing additional you need to do to authenticate the Collector process. Simply run the Collector on your instance, and it will inherit these permissions.
**GKE / Workload Identity**
On GKE clusters with Workload Identity enabled (including GKE Autopilot), follow [the steps to configure a Workload Identity ServiceAccount in your cluster](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) (if you do not already have one). Then, deploy the Collector as you would with any other workload, setting the `serviceAccountName` field in the Collector Pod’s `.spec` to the WI-enabled ServiceAccount.
In non-WI clusters, you can use the GCP service account associated with the node the same way as in the instructions for GCE instances above.
**Non-GCP (AWS, Azure, on-prem, etc.) or alternative service accounts**
In non-GCP environments, a [service account key](https://cloud.google.com/iam/docs/keys-create-delete#iam-service-account-keys-create-console) or credentials file is required. The exporter will automatically look for this file using the `GOOGLE_APPLICATION_CREDENTIALS` environment variable or, if that is unset, one of the [other known locations](https://cloud.google.com/docs/authentication/application-default-credentials). Note that when using this approach, you may need to explicitly set the `project` option in the exporter’s config.
When running the Collector in a Docker container, a credentials file can be passed to the container via volume mounts and environment variables at runtime like so:
```
docker run \
--volume ~/service-account-key.json:/etc/otelcol-contrib/key.json \
--volume $(pwd)/config.yaml:/etc/otelcol-contrib/config.yaml \
--env GOOGLE_APPLICATION_CREDENTIALS=/etc/otelcol-contrib/key.json \
--expose 4317 \
--expose 55681 \
--rm \
otel/opentelemetry-collector-contrib
```
**Using `gcloud auth application-default login`**
Using [`gcloud auth application-default login`](https://cloud.google.com/docs/authentication/application-default-credentials) to authenticate is not recommended for production use. Instead, it’s best to use a GCP service account through one of the methods listed above. The `gcloud auth` command can be useful for development and testing on a user account, and authenticating with it follows the same approach as the service account key method above.
These instructions are to get you up and running quickly with the GCP exporter in a local development environment. We'll also point out alternatives that may be more suitable for CI or production.
1. **Obtain a Collector binary.** Pull a binary or Docker image for the
OpenTelemetry contrib collector which includes the GCP exporter plugin
through one of the following:
* Download a [binary or package of the OpenTelemetry
Collector Contrib](https://github.com/open-telemetry/opentelemetry-collector-releases/releases)
that is appropriate for your platform, and includes the Google Cloud
exporter.
* Pull a Docker image with `docker pull otel/opentelemetry-collector-contrib`
* Create your own main package in Go, that pulls in just the plugins you need.
* Use the [OpenTelemetry Collector
Builder](https://github.com/open-telemetry/opentelemetry-collector-builder)
to generate the Go main package and `go.mod`.
2. **Create a configuration file `config.yaml`.** The example below shows a minimal recommended configuration that receives OTLP and sends data to GCP, in addition to verbose logging to help understand what is going on. It uses application default credentials (which we will set up in the next step).
Note that this configuration includes the recommended `memory_limiter` plugins, which avoid high latency for reporting telemetry, and ensure that the collector itself will stay stable (not run out of memory) by dropping telemetry if needed.
```yaml theme={null}
receivers:
otlp:
protocols:
grpc:
http:
exporters:
googlecloud:
log:
default_log_name: opentelemetry.io/collector-exported-log
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 65
spike_limit_percentage: 20
resource_detection:
detectors: [gcp]
timeout: 10s
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter]
exporters: [googlecloud]
metrics:
receivers: [otlp]
processors: [memory_limiter]
exporters: [googlecloud]
logs:
receivers: [otlp]
processors: [memory_limiter]
exporters: [googlecloud]
```
3. **Set up credentials.**
1. Enable billing in your GCP project.
2. Enable the Cloud Metrics and Cloud Trace APIs.
3. Ensure that your user GCP user has (at minimum) `roles/monitoring.metricWriter` and `roles/cloudtrace.agent`. You can learn about [metric-related](https://cloud.google.com/monitoring/access-control) and [trace-related](https://cloud.google.com/trace/docs/iam) IAM in the GCP documentation.
4. Obtain credentials using one of the methods in the [Authenticating
section](#prerequisite-authenticating) above.
4. **Run the collector.** The following runs the collector in the foreground, so please execute it in a separate terminal.
```sh theme={null}
./otelcol-contrib --config=config.yaml
```
Alternatives
If you obtained OS-specific packages or built your own binary in step 1, you'll need to follow the appropriate conventions for running the collector.
5. **Gather telemetry.** Run an application that can submit OTLP-formatted metrics and traces, and configure it to send them to `127.0.0.1:4317` (for gRPC) or `127.0.0.1:55681` (for HTTP).
Alternatives
* Set up the host metrics receiver, which will gather telemetry from the host without needing an external application to submit telemetry.
* Set up an application-specific receiver, such as the Nginx receiver, and run the corresponding application.
* Set up a receiver for some other protocol (such Prometheus, StatsD, Zipkin or Jaeger), and run an application that speaks one of those protocols.
6. **View telemetry in GCP.** Use the GCP [metrics explorer](https://console.cloud.google.com/monitoring/metrics-explorer) and [trace overview](https://console.cloud.google.com/traces) to view your newly submitted telemetry.
## Configuration reference
The following configuration options are supported:
* `project` (default = Fetch from Credentials): GCP project identifier.
* `destination_project_quota` (optional, default = false): Counts quota against the project to which the data is sent (as opposed to the project associated with the Collector's service account. For example, when setting `project_id` or using [multi-project export](#multi-project-exporting).
* `user_agent` (default = `collector description/version os/arch`, i.e. `opentelemetry-collector-contrib/v0.139.0 linux/amd64`): Override the user agent string sent on requests to Cloud Monitoring (currently only applies to metrics). Specify `{{version}}` to include the application version number.
* `timeout` (default = `12s`) The timeout for requests to Google Cloud Platform APIs, specified in Go Time Duration format.
* `impersonate` (optional): Configuration for service account impersonation
* `target_principal`: TargetPrincipal is the email address of the service account to impersonate.
* `subject`: (optional) Subject is the sub field of a JWT. This field should only be set if you wish to impersonate as a user. This feature is useful when using domain wide delegation.
* `delegates`: (default = \[]) Delegates are the service account email addresses in a delegation chain. Each service account must be granted roles/iam.serviceAccountTokenCreator on the next service account in the chain.
* `metric` (optional): Configuration for sending metrics to Google Cloud Monitoring.
* `prefix` (default = `workload.googleapis.com`): The prefix to add to metrics.
* `endpoint` (default = `monitoring.googleapis.com`): Endpoint where metric data is going to be sent to.
* `use_insecure` (default = false): If true, disables gRPC client transport security. Only has effect if Endpoint is not "".
* `compression` (optional, supported values: \[`gzip`]): Compression format for Metrics gRPC requests. Defaults to no compression.
* `grpc_pool_size` (optional): Sets the size of the connection pool in the GCP client. Defaults to a single connection.
* `known_domains` (default = \[googleapis.com, kubernetes.io, istio.io, knative.dev]): If a metric belongs to one of these domains it does not get a prefix.
* `skip_create_descriptor` (default = false): If set to true, do not send metric descriptors to Google Cloud Monitoring.
* `instrumentation_library_labels` (default = true): If true, the exporter will copy the OTLP `InstrumentationScope.Name` to a label `instrumentation_source` and `InstrumentationScope.Version` to a label `instrumentation_version` labels on metrics.
* `service_resource_labels` (default = true): If true, the exporter will copy the Semantic Conventions `service.name`, `service.namespace`, and `service.instance.id` from OTLP Resource Attributes into the Google Cloud Monitoring timeseries metric labels. These labels will be the same as the Semantic Conventions with `.` replaced by `_`.
* `create_metric_descriptor_buffer_size` (default = 10): Buffer size for the channel which asynchronously calls CreateMetricDescriptor.
* `resource_filters` (default = \[]): If provided, resource attributes matching any filter will be included in metric labels. Can be defined by `prefix`, `regex`, or `prefix` AND `regex`.
* `prefix`: Match resource keys by prefix.
* `regex`: Match resource keys by regex.
* `cumulative_normalization` (default = true): If true, normalizes cumulative metrics without start times or with explicit reset points by subtracting subsequent points from the initial point. It is enabled by default. Since it caches starting points, it may result in increased memory usage.
* `sum_of_squared_deviation` (default = false): If true, enables calculation of an estimated sum of squared deviation. It is an estimate, and is not exact.
* `create_service_timeseries` (default = false): If true, this will send all timeseries using `CreateServiceTimeSeries`. Implicitly, this sets `skip_create_descriptor` to true.
* `experimental_wal` (default = \[]): If provided, enables use of a write ahead
log for time series requests.
* `directory` (default = `./`): Path to local directory for WAL file.
* `max_backoff` (default = `1h`): Max duration to retry requests on network errors (`UNAVAILABLE` or `DEADLINE_EXCEEDED`).
* `trace` (optional): Configuration for sending traces to Google Cloud Trace.
* `endpoint` (default = `cloudtrace.googleapis.com`): Endpoint where trace data is going to be sent to.
* `use_insecure` (default = false): If true, disables gRPC client transport security. Only has effect if Endpoint is not "".
* `grpc_pool_size` (optional): Sets the size of the connection pool in the GCP client. Defaults to a single connection.
* `attribute_mappings` (optional): AttributeMappings determines how to map from OpenTelemetry attribute keys to Google Cloud Trace keys. By default, it changes `http` and `service` keys so that they appear more prominently in the UI.
* `key`: The OpenTelemetry attribute key
* `replacement`: The replacement attribute sent to Google Cloud Trace
* `log` (optional): Configuration for sending logs to Google Cloud Logging.
* `endpoint` (default = `logging.googleapis.com`): Endpoint where log data is going to be sent to.
* `use_insecure` (default = false): If true, disables gRPC client transport security. Only has effect if Endpoint is not "".
* `compression` (optional, Supported values: \[`gzip`]): Compression format for Logs gRPC requests. Defaults to no compression.
* `grpc_pool_size` (optional): Sets the size of the connection pool in the GCP client. Defaults to a single connection.
* `default_log_name` (optional): Defines a default name for log entries. If left unset, and a log entry does not have the `gcp.log_name` attribute set, the exporter will return an error processing that entry.
* `resource_filters` (default = \[]): If provided, Resource Attributes matching any filter will be included in log entry labels. Can be defined by `prefix`, `regex`, or `prefix` AND `regex`.
* `prefix`: Match resource keys by prefix.
* `regex`: Match resource keys by regex.
* `sending_queue` (optional): Configuration for how to buffer data before sending. Note: The `sending_queue` is provided (and documented) by the [Exporter Helper](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/exporterhelper#sending-queue)
Beyond standard YAML configuration as outlined in the section above,
exporters that leverage the net/http package (all do today) also respect the
following proxy environment variables:
* HTTP\_PROXY
* HTTPS\_PROXY
* NO\_PROXY
If set at Collector start time then exporters, regardless of protocol,
will or will not proxy traffic as defined by these environment variables.
### Monitored Resources
For metrics and logs, this exporter maps the OpenTelemetry Resource to a Google
Cloud [Logging](https://cloud.google.com/logging/docs/api/v2/resource-list) or
[Monitoring](https://cloud.google.com/monitoring/api/resources) Monitored Resource.
The complete mapping logic can be found in [resourcemapping.go](https://github.com/GoogleCloudPlatform/opentelemetry-operations-go/blob/main/internal/resourcemapping/resourcemapping.go).
That may be the most helpful reference if you want to map to a specific monitored
resource.
#### On GCP
If running on GCP, using the GCP resource detector, as shown above, will populate
the resource attributes required to map to the appropriate monitored resource.
#### Off GCP
If you are not running on GCP, you still need to choose a [GCP zone or
region](https://cloud.google.com/compute/docs/regions-zones) to send telemetry to
by setting `cloud.availability_zone` or `cloud.region`. In addition, you should use the detector associated with other cloud providers, if applicable.
If running on Kubernetes, it is recommended to additionally set `k8s.pod.name`,
`k8s.namespace.name`, and `k8s.container.name` using the `k8sattributes` processor.
If you are getting "duplicate timeseries encountered" errors, it is likely because
you are missing a required resource attribute, causing a metric from two different
instances of an application to end up with the same monitored resource.
### Preventing metric label collisions
The metrics exporter can add metric labels to timeseries, such as when setting
`metric.service_resource_labels`, `metric.instrumentation_library_labels` (both
on by default), or when using `metric.resource_filters` to convert resource
attributes to metric labels.
However, if your metrics already contain any of these labels they will fail to
export to Google Cloud with a `Duplicate label key encountered` error. Such
labels from the default features above include:
* `service_name`
* `service_namespace`
* `service_instance_id`
* `instrumentation_source`
* `instrumentation_version`
*(Note that these are the sanitized versions of OpenTelemetry attributes, with `.` replaced by `_` to be compatible with Cloud Monitoring. For example, `service_name` comes from the [`service.name` resource attribute](https://github.com/open-telemetry/opentelemetry-specification/blob/dc78006c12d9767fd2e35b691706c7572a76fd43/specification/resource/semantic_conventions/README.md#service).)*
To prevent this, it's recommended to use the [transform processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/c7bd50ce773e66be327ef7618775a884a774e5d1/processor/transformprocessor) in your collector config to rename existing metric labels to preserve them, for example:
```yaml theme={null}
processors:
transform:
metric_statements:
- context: datapoint
statements:
- set(attributes["exported_service_name"], attributes["service_name"])
- delete_key(attributes, "service_name")
- set(attributes["exported_service_namespace"], attributes["service_namespace"])
- delete_key(attributes, "service_namespace")
- set(attributes["exported_service_instance_id"], attributes["service_instance_id"])
- delete_key(attributes, "service_instance_id")
- set(attributes["exported_instrumentation_source"], attributes["instrumentation_source"])
- delete_key(attributes, "instrumentation_source")
- set(attributes["exported_instrumentation_version"], attributes["instrumentation_version"])
- delete_key(attributes, "instrumentation_version")
```
**Note** It is not recommended to use these transformations with the googlecloud
exporter in a logging or trace pipeline.
The same method can be used for any resource attributes being filtered to metric
labels, or metric labels which might collide with the GCP monitored resource
used with resource detection.
Keep in mind that your conflicting attributes may contain dots instead of
underscores (eg, `service.name`), but these will still collide once all
attributes are normalized to metric labels. In this case you will need to update
the collector config above appropriately.
### Logging Example
The logging exporter processes OpenTelemetry log entries and exports them to GCP Cloud Logging. Logs can be collected using one
of the opentelemetry-collector-contrib log receivers, such as the [filelogreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver).
Log entries must contain any Cloud Logging-specific fields as a matching OpenTelemetry attribute (as shown in examples from the
[logs data model](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#google-cloud-logging)).
These attributes can be parsed using the various [log operators](../../pkg/stanza/docs/operators/README.md#what-operators-are-available) available upstream.
For example, the following config parses the [HTTPRequest field](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#HttpRequest) from Apache log entries saved in `/var/log/apache.log`.
It also parses out the `timestamp` and inserts a non-default `log_name` attribute and GCP [MonitoredResource](https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource) attribute.
```yaml theme={null}
receivers:
file_log:
include: [ /var/log/apache.log ]
start_at: beginning
operators:
- id: http_request_parser
type: regex_parser
regex: '(?m)^(?P[^ ]*) (?P[^ ]*) (?P[^ ]*) \[(?P