Journal Files

The Journal Files Source collects data from systemd’s centralized logging, which is called the journald service. This service is a centralized location for all messages logged by different components in a systemd-enabled Linux system. This includes messages from kernel, boot, syslog, or other services. This Source is currently configured to configured to collect data from the user journal, as system is privileged. This Source is disabled by default.

Type: System | TLS Support: N/A | Event Breaker Support: No

The Journal Files Source is available on Cribl.Cloud hybrid Workers, but not on Cribl-managed Cribl.Cloud Workers.

Discovering and Filtering Journal Files to Monitor​

To determine all the names of all journals to read, the Journal Files Source runs a discovery procedure at a configurable Polling interval. The Source then applies an Allowed list to filter the initial list down into its final form. Then, for each journal on the list, the Source compares current state with previously stored state. This comparison determines whether the Journal Files Source will actually stream a given journal for a given polling interval. The Source then runs the files through the Filter Rules which allows you to set additional criteria.

Cribl Edge ships with a single, disabled instance of the Journal Files Source with a default configuration. The search path is configured to point to $CRIBL_EDGE_FS_ROOT/var/log/journal/$MACHINE_ID and defaults to streaming the primary system.journal it finds there. $MACHINE_ID is an environment variable that you can insert into the Source’s search path. If you use the environment variable but don’t assign a value, the Source will look up the local host machine id from /etc/machine-id and use that value.

Currently, this Source does not support all of the compression options that the journald service supports. The parser this Source uses to read the files can handle LZ4 and ZSTD compression. When the parser encounters compressed fields it can’t unpack, an error message emits in the logs. Below is an example of the error message:

Error reading journald event, dropping offset=31678152 evt="Mar 02 21:29:54 goats-xps15 multipassd: undefined"

Also, this Source does not support the new COMPACT binary file format. The logs will contain an error message when the parser encounters this type of file.

Configuring a Journal Files Source​

From the top nav, click Manage, then select a Fleet to configure. Next, you have two options:

To configure via the graphical QuickConnect UI, click Routing > QuickConnect (Stream) or Collect (Edge). Next, click Add Source at left. From the resulting drawer’s tiles, select [System and Internal >] Journal Files. Next, click either Add Destination or (if displayed) Select Existing. The resulting drawer will provide the options below.

Or, to configure via the Routing UI, click Data > Sources (Stream) or More > Sources (Edge). From the resulting page’s tiles or left nav, select [System and Internal >] Journal Files. Next, click Add Source to open a New Source modal that provides the options below.

General Settings​

Enabled: Toggle to Yes to enable the Source.

Input ID: Enter a unique name to identify this File Monitor Source definition. If you clone this Source, Cribl Edge will add -CLONE to the original Input ID.

Search path: Specify the directory to search journals on. Accepts environment variables. The default value for this field varies depending on the product:

  • In Cribl Edge, it defaults to $CRIBL_EDGE_FS_ROOT/var/log/journal/$MACHINE_ID.
  • In Cribl Stream, it defaults to /var/log/journal/$MACHINE_ID.

Journal allowlist: The full path of each discovered journal is matched against this wildcard list. Defaults to system.

Optional Settings​

Polling interval: How often, in seconds, to collect metrics. If not specified, defaults to 10 seconds.

Filter Rules: Optionally, specify which journal objects to allow. Events are generated if all the rules’ expressions evaluate to true, or if no rules are specified.

  • Filter expression: JavaScript expression to filter Journal objects. Returns true to include it.

  • Description: Optional description of the rule.

    The default Filter expression is severity <= 4 with a Description of Allow events having 'emergency', 'alert', 'critical', 'error', or 'warning' priority.

    The screenshot below shows other examples you can use:

Filter expressions
Filter expressions

Current boot only: Skip events that are not part of the current boot session.

Max age duration: Optionally, specify a maximum age of journal objects to stream. This Source will filter events with timestamps newer than the configured duration. Enter a duration in a format like 30s, 4h, 3d, or 1w. Defaults to an empty field, which applies no age filters.

Tags: Optionally, add tags that you can use to filter and group Sources in Cribl Edge’s Manage Sources page. These tags aren’t added to processed events. Use a tab or hard return between (arbitrary) tag names.

Processing Settings​

Fields​

In this section, you can add Fields to each event, using Eval-like functionality.

Name: Field name.

Value: JavaScript expression to compute field’s value, enclosed in quotes or backticks. (Can evaluate to a constant.)

Pre-Processing​

In this section’s Pipeline drop-down list, you can select a single existing Pipeline to process data from this input before the data is sent through the Routes.

Advanced Settings​

Environment: If you’re using GitOps, optionally use this field to specify a single Git branch on which to enable this configuration. If empty, the config will be enabled everywhere.

Connected Destinations​

Select Send to Routes to enable conditional routing, filtering, and cloning of this Source’s data via the Routing table.

Select QuickConnect to send this Source’s data to one or more Destinations via independent, direct connections.

In Cribl Edge, this Source defaults to QuickConnect. While, in Cribl Stream, it defaults to Send to Routes.