> ## Documentation Index > Fetch the complete documentation index at: https://otel.fyi/llms.txt > Use this file to discover all available pages before exploring further. # Journald > OpenTelemetry receiver for Journald # Journald Receiver ![Status](https://img.shields.io/badge/status-alpha-red) **Available in:** `contrib`, `k8s` **Maintainers:** [@belimawr](https://github.com/belimawr), [@namco1992](https://github.com/namco1992) **Source:** [opentelemetry-collector-contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/journaldreceiver) ## Supported Telemetry ![Logs](https://img.shields.io/badge/logs-alpha-blue) ## Overview ## Prerequisites The journald receiver has specific requirements that must be met: ### journalctl Binary Requirement The journald receiver **requires the journalctl binary** to be available in the container or on the system. This is because the receiver uses journalctl to read logs from the systemd journal. The receiver does not implement its own journal reading logic and relies on journalctl for this functionality. If running the collector in a container, the journalctl binary should match the host system, and this receiver should be configured with the `root_path`/`journalctl_path` options so that the journalctl from the host is used. ### Permissions * The collector's user must have sufficient permissions to access the journal through `journalctl` * In containerized environments, this typically requires root privileges and specific capabilities (see [Kubernetes Security](#kubernetes-security-requirements)) ## Configuration | Field | Default | Description | | ----------------------------------- | ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `directory` | `/run/log/journal` or `/run/journal` | A directory containing journal files to read entries from. Relative to `root_path`. | | `files` | | A list of journal files to read entries from. Relative to `root_path`. | | `start_at` | `end` | At startup, where to start reading logs from the file. Options are beginning or end | | `units` | | A list of units to read entries from. See [Multiple filtering options](#multiple-filtering-options) examples. | | `identifiers` | | Filter output by message identifiers (`SYSTEMD_IDENTIFIER`). See [Multiple filtering options](#multiple-filtering-options) examples. | | `matches` | | A list of matches to read entries from. See [Matches](#matches) and [Multiple filtering options](#multiple-filtering-options) examples. | | `priority` | `info` | Filter output by message priorities or priority ranges. See [Multiple filtering options](#multiple-filtering-options) examples. | | `grep` | | Filter output to entries where the MESSAGE= field matches the specified regular expression. See [Multiple filtering options](#multiple-filtering-options) examples. | | `dmesg` | 'false' | Show only kernel messages. This shows logs from current boot and adds the match `_TRANSPORT=kernel`. See [Multiple filtering options](#multiple-filtering-options) examples. | | `storage` | none | The ID of a storage extension to be used to store cursors. Cursors allow the receiver to pick up where it left off in the case of a collector restart. If no storage extension is used, the receiver will manage cursors in memory only. | | `all` | 'false' | If `true`, very long logs and logs with unprintable characters will also be included. | | `namespace` | | Will query the given namespace. See man page [`systemd-journald.service(8)`](https://www.man7.org/linux/man-pages/man8/systemd-journald.service.8.html#JOURNAL_NAMESPACES) for details. | | `convert_message_bytes` | 'false' | If `true` and if the `MESSAGE` field is read [as an array of bytes](https://github.com/systemd/systemd/blob/main/docs/JOURNAL_EXPORT_FORMATS.md#journal-json-format), the array will be converted to string. | | `merge` | 'false' | If `true`, read from all available journals, including remote ones. | | `retry_on_failure.enabled` | `false` | If `true`, the receiver will pause reading a file and attempt to resend the current batch of logs if it encounters an error from downstream components. | | `retry_on_failure.initial_interval` | `1 second` | Time to wait after the first failure before retrying. | | `retry_on_failure.max_interval` | `30 seconds` | Upper bound on retry backoff interval. Once this value is reached the delay between consecutive retries will remain constant at the specified value. | | `retry_on_failure.max_elapsed_time` | `5 minutes` | Maximum amount of time (including retries) spent trying to send a logs batch to a downstream consumer. Once this value is reached, the data is discarded. Retrying never stops if set to `0`. | | `root_path` | | Chroot to use when executing the journalctl command. Must be an absolute path or empty. When empty (default), no chroot is used when executing journalctl. | | `journalctl_path` | `journalctl` | journalctl command to execute. Relative to `root_path`. Must be an absolute path if `root_path` is non-empty. See below for more details | | `operators` | \[] | An array of [operators](../../pkg/stanza/docs/operators/README.md#what-operators-are-available). See below for more details | ### Operators Each operator performs a simple responsibility, such as parsing a timestamp or JSON. Chain together operators to process logs into a desired format. * Every operator has a `type`. * Every operator can be given a unique `id`. If you use the same type of operator more than once in a pipeline, you must specify an `id`. Otherwise, the `id` defaults to the value of `type`. * Operators will output to the next operator in the pipeline. The last operator in the pipeline will emit from the receiver. Optionally, the `output` parameter can be used to specify the `id` of another operator to which logs will be passed directly. * Only parsers and general purpose operators should be used. ### Example Configurations #### Minimal configuration The following configuration is the minimal configuration to read journald logs: ```yaml theme={null} receivers: journald: ``` will be passed to journalctl as the following arguments: `journalctl ... --priority info`. This will read the 10 most recent entries and any subsequent entry. `--priority info` is the default priority, the following examples will omit it for simplicity. #### Cursor tracking ```yaml theme={null} receivers: journald: storage: file_storage/journald extensions: file_storage/journald: directory: . service: extensions: [file_storage/journald] ``` If you stop and start the otel collector, only new entries will be read. #### Reading from the beginning ```yaml theme={null} receivers: journald: start_at: beginning ``` will be passed to journalctl as the following arguments: `journalctl ... --no-tail`. This will read all messages from the current boot. #### Units ```yaml theme={null} receivers: journald: directory: /run/log/journal units: - ssh - kubelet - docker - containerd priority: info ``` #### Matches The following configuration: ```yaml theme={null} - type: journald_input matches: - _SYSTEMD_UNIT: ssh - _SYSTEMD_UNIT: kubelet _UID: "1000" ``` will be passed to `journalctl` as the following arguments: `journalctl ... _SYSTEMD_UNIT=ssh + _SYSTEMD_UNIT=kubelet _UID=1000`, which is going to retrieve all entries which match at least one of the following rules: * `_SYSTEMD_UNIT` is `ssh` * `_SYSTEMD_UNIT` is `kubelet` and `_UID` is `1000` #### Multiple filtering options In case of using multiple following options, conditions between them are logically `AND`ed and within them are logically `OR`ed: ```text theme={null} ( dmesg ) AND ( priority ) AND ( units[0] OR units[1] OR units[2] OR ... units[U] ) AND ( identifier[0] OR identifier[1] OR identifier[2] OR ... identifier[I] ) AND ( matches[0] OR matches[1] OR matches[2] OR ... matches[M] ) AND ( grep ) ``` Consider the following example: ```yaml theme={null} - type: journald_input matches: - _SYSTEMD_UNIT: ssh - _SYSTEMD_UNIT: kubelet _UID: "1000" units: - kubelet - systemd priority: info identifiers: - systemd ``` The above configuration will be passed to `journalctl` as the following arguments `journalctl ... --priority=info --unit=kubelet --unit=systemd --identifier=systemd _SYSTEMD_UNIT=ssh + _SYSTEMD_UNIT=kubelet _UID=1000`, which is going to effectively retrieve all entries which matches the following set of rules: * `_PRIORITY` is `6`, and * `_SYSTEMD_UNIT` is `kubelet` or `systemd`, and * `SYSLOG_IDENTIFIER` `systemd`, and * entry matches at least one of the following rules: * `_SYSTEMD_UNIT` is `ssh` * `_SYSTEMD_UNIT` is `kubelet` and `_UID` is `1000` ## Performance Considerations ### start\_at parameter The `start_at` configuration parameter has significant performance implications: * **`end` (default - recommended for production)**: Reads only new entries after collector start * Minimal startup impact * No historical log replay * **`beginning`**: Reads entire journal history (potentially GBs of logs) * Use only for testing or initial backfill * Can cause 10-60 second startup delay * High CPU/memory usage during catchup ### Unit Filtering Impact * **Unfiltered**: Reads all systemd units * **Filtered**: Reads only specified units ## Setup and Deployment ### Journal Location in Container Environments Journal file locations differ between traditional systems and containerized environments: **Traditional Systems:** * Persistent journal: `/var/log/journal/` * Requires `Storage=persistent` in journald.conf **Container Platforms (kind, minikube, k3s, most Kubernetes clusters):** * Volatile journal: `/run/log/journal/` * Default `Storage=volatile` or `Storage=auto` with missing `/var/log/journal` **Discovery Command:** ```bash theme={null} journalctl --header | grep "File path" ``` ### Docker & Kubernetes When running otelcol in a container, you need: 1. Root permissions (journal files are root-owned) 2. The journal directory mounted (usually `/run/log/journal` in containerized environments) 3. The `journalctl` binary available #### Option 1: Bundle journalctl in your image (Recommended) Build a collector image that includes journalctl. See the example `Dockerfile` in [`examples/container`](examples/container/README.md) with step-by-step instructions for Kubernetes deployments. This works well if your hosts run similar systemd versions. #### Option 2: Use host's journalctl via chroot Some collector containers need to work across a variety of arbitrary hosts that may be running mutually incompatible versions of journalctl, making it difficult to select a single version of journalctl to bundle into the container. One way to solve this problem is to ensure that the collector running in the container invokes the exact same journalctl that was used to write the logs in the first place. To achieve this, you can mount the host's rootfs to the container and then configure the receiver to run the host's journalctl in a chroot for that mount. You can pass `-v /:/host` to `docker run` to mount the host's rootfs: ``` docker run -v /:/host otel/opentelemetry-collector-contrib ``` Then, you can configure the receiver with `root_path` to use that mount as a chroot for journalctl. Due to a [Go issue](https://github.com/golang/go/issues/39341), running executables from \$PATH does not work well in a chroot, so you must also use `journalctl_path` to configure a full path to `journalctl` inside of the chroot: ```yaml theme={null} receivers: journald: root_path: /host journalctl_path: /usr/bin/journalctl ``` You also need to ensure the user running the collector can run the journalctl from the chroot, one way to do this is to run the collector as root: ``` docker run -v /:/host --user 0 otel/opentelemetry-collector-contrib ``` ### Linux packaging When installing otelcol as a linux package, you will most likely need to add the `otelcol-contrib` or `otel` user to the `systemd-journal` group. The exact user and group might vary depending on your package and linux distribution of choice. You can test if the user has sufficient permissions by running something like (you might need to adjust according to available shell and opentelemetry user) ```sh theme={null} sudo su -s /bin/bash -c 'journalctl --lines 5' otelcol-contrib ``` if the permissions are set correctly you will see some logs, otherwise a clear error message. ## Troubleshooting ### Kubernetes Security Requirements **Minimum Requirements:** * Run as root (`runAsUser: 0`) - journal files are root-owned * Read access to `/run/log/journal` (requires root permission level) * Required capabilities: * `CAP_DAC_READ_SEARCH`: Read any file regardless of permissions * `CAP_SYS_PTRACE`: Required by journalctl for some operations **Recommended security context:** ```yaml theme={null} securityContext: runAsUser: 0 capabilities: add: - DAC_READ_SEARCH - SYS_PTRACE ``` ### Common Errors and Solutions | Error Message | Cause | Solution | | ------------------------------------------ | -------------------------------------------- | -------------------------------------------------------------------------------------------------- | | `journalctl: executable file not found` | systemd not installed in container | Install systemd package in your container image | | `Journal file uses an unsupported feature` | systemd version mismatch | Use Ubuntu 24.04, Debian 12, or newer systemd version | | `permission denied` reading journal | Insufficient permissions | Add `CAP_DAC_READ_SEARCH` capability and run as root | | `directory "/var/log/journal" is empty` | Wrong journal path for container environment | Change to `/run/log/journal` (see [Journal Location](#journal-location-in-container-environments)) | | No logs appearing | Various causes | See diagnostic commands below | | `Failed to get journal file list` | Incorrect mount or permissions | Verify volume mount and security context | ### Diagnostic Commands ```bash theme={null} # 1. Check journal location on the host ls -la /var/log/journal /run/log/journal journalctl --header | grep "File path" # 2. Verify journalctl works inside the container kubectl exec -it -- journalctl -n 20 # 3. Check journalctl version kubectl exec -it -- journalctl --version # 4. Test reading from mounted directory kubectl exec -it -- journalctl -D /run/log/journal --priority=info -n 10 # 5. Check collector logs for errors kubectl logs | grep -i "journald\|error" # 6. Verify volume mounts kubectl exec -it -- ls -la /run/log/journal # 7. Check pod security context kubectl get pod -o jsonpath='{.spec.containers[0].securityContext}' # 8. Verify capabilities (from within pod) kubectl exec -it -- grep Cap /proc/self/status ``` ## Configuration ### Example Configuration ```yaml theme={null} journald: units: - ssh priority: info directory: /run/log/journal ``` *** *Last generated: 2026-07-06*