Securing Communications

This page outlines how to protect Cribl Stream Leader to Worker Nodes communications, using an existing TLS/SSL certificate and key.

Cribl Stream expects certificates and keys to be formatted in privacy-enhanced mail (.pem) format. To generate a self-signed certificate and corresponding key, see Securing Cribl Stream.

Importing Certificate and Key​

To use your certificate and key to prepare secure communications between Workers/Edge Nodes and the Leader:

1. Navigate to the Leader’s Settings > Global Settings > Security > Certificates > New Certificates modal.
2. Open your TLS/SSL certificate file. (The self-signed certificate example used the placeholder name myCert.pem.)
3. Copy the file’s contents to your clipboard.
4. Paste the file’s contents into the modal’s Certificate field.

You can skip the preceding three steps: Just drag/drop your .pem file from your filesystem into the Certificate field, or upload it using the button at the field’s upper right.

5. Open your private key file. (The self-signed certificate example used the placeholder name myKey.pem.)
6. Copy the file’s contents to your clipboard.
7. Paste the clipboard contents into the same Leader modal’s Private key field.

Here again, you can skip the preceding three steps by dragging/dropping or uploading the .pem file from your filesystem.

8. Fill in the Name and Description fields.
9. If you’ve uploaded a self-signed certificate, just Save it now.
10. If your private key is encrypted, fill in the modal’s Passphrase with the corresponding key.
    (You can paste the key’s contents, or you can drag/drop or upload the key file.)
11. If you’re uploading a certificate signed by an external certificate authority – e.g., a downloaded Splunk Cloud certificate – import the chain into the CA certificate field before saving the cert. For details, see Obtain the Certificate Chain (TLS/SSL).

Connecting to the Leader Securely​

You can configure secure communication between your Leader and Worker Nodes using the UI, the instance.yml config file, or environment variables.

Using the UI​

To set up secure communication via the UI, you configure first the Worker Group, then the Worker Node, then the Leader.

Worker Group Setup​

For each Worker Group whose Worker Nodes you want to secure:

1. Open your TLS/SSL certificate file, and copy its contents to your clipboard. (This can be the same certificate you uploaded to the Leader, or a different cert.)
2. Select Manage, then select the Worker Group you want to configure.
3. Select Group Settings or Fleet Settings (upper right).
4. From the left nav, select Security > Certificates > TLS.
5. Click Add Certificate.
6. In the resulting New Certificates modal, add the cert’s contents to the Certificate field.
   (You can paste the cert file’s contents, or you can drag/drop or upload the .pem file.)
7. As you did on the Leader, also insert your Private key, and (as needed) your Passphrase and CA certificate.
8. Click Save.
9. Commit and Deploy the Worker Group’s new configuration, including the new cert.
10. Repeat the preceding steps on each Worker Group.

Worker Node Setup​

For each Worker Node that you want to secure:

1. Enable the Leader’s UI access to each desired Worker Node (Stream, Edge).
2. Tunnel through from the Leader to a Worker Node’s UI.
3. Navigate to this Worker Node’s Worker Settings (upper right) > System > Distributed Settings > TLS Settings.
4. Toggle Enable Server TLS to Yes.
   This will expose the remaining TLS settings.
5. From the Certificate name drop-down, select the certificate you uploaded to the parent Worker Group.
   This will prefill all the required fields. (See all deployed certificates at the left nav’s Security > Certificates link.)
6. Click Save.
   The Worker will be unavailable during a short lag, while it restarts with the new configuration.
7. Repeat the preceding steps on each Worker Node.

Leader Setup​

Next, return to the Leader’s UI:

1. Select Settings > Global Settings> System > Distributed Settings > TLS Settings.
2. Toggle Enable server TLS to Yes.
3. In the Certificate name drop-down, select an existing Certificate. This will auto-populate the corresponding cert fields.
4. Click Save.

After you’ve enabled TLS on the Leader, generating bootstrap scripts to add or update Worker Nodes will automatically prepend https:// to the Leader’s URL.

Using YAML Config File​

You can also configure the Leader’s $CRIBL_HOME/local/_system/instance.yml file to ensure that TLS is enabled. Here’s the relevant section:

distributed:
  mode: managed-edge
  master:
    host: <hostname>
    port: 4200
    authToken: <token>
    tls:
      disabled: false
      rejectUnauthorized: false
      requestCert: false
    resiliency: none
  group: default_fleet

After you’ve enabled TLS on the Leader, generating bootstrap scripts to add or update Worker Nodes will automatically prepend https:// to the Leader’s URL.

Using Environment Variables​

Another way to set up secure communications between Worker Nodes and the Leader is via environment variables (Stream, Edge).

If you deploy your Worker Nodes in a container, you can enable encrypted TLS communications with the Leader by configuring the CRIBL_DIST_MASTER_URL with the tls: protocol. This will override the default setting in instance.yml. Here’s the format:

CRIBL_DIST_LEADER_URL=tls://<authToken>@leader:4200

Configuring TLS Mutual Authentication​

Once you have configured the Leader for encrypted TLS communication, you can implement client certificate exchange to enable mutual authentication. This allows Cribl Stream to permit only explicitly authorized clients, which hold valid certificates, to connect to the Leader.

When a client certificate is presented to a Leader, two things happen:

1. The Leader validates the client certificate presented to Cribl Stream. The Validate Client Certs setting is optional, but highly recommended. When enabled, the Leader checks the certificate against the trust store, to see if it has been signed by a valid certificate authority (CA).

The Leader checks the list of certificates in the CA Certificates Path box first (if populated), then against the list of built-in system certificates.

1. The Leader checks whether the Common Name (CN) matches the regular expression in the configuration. Cribl Stream’s default is to accept any value in the Common Name field. You can customize this as needed.

Within the Common Name, we validate against the value after the CN=string. If your Common Name is CN=logstream.worker, you would enter logstream\.worker in the Common Name field – including the backslash, because the value entered is a regular expression.

Limitations on TLS Mutual Auth​

When configuring TLS mutual authentication on Worker Nodes, make sure you place your certificates into a separate directory outside of $CRIBL_HOME. If you place the certificates inside $CRIBL_HOME, they’ll be removed when the next config bundle is deployed from the Leader.

Similarly, you can’t bootstrap Worker Nodes with mutual authentication already populated. To bootstrap Worker Nodes, you supply only the shared authentication token. Certificates should be viewed as two-factor authentication; so placing the certificates in the config bundle defeats the purpose of two-factor authentication.

Using the UI​

TLS mutual authentication in Cribl.Cloud is configured separately for each Source.

Below is a sample configuration on the Leader:

In the above configuration, note:

• You can set both the Minimum TLS version and Maximum TLS version.

• The Common Name regex contains an additional check for \d+. This allows checking for the format: logstream1.worker, logstream2.worker, logstream3.worker, etc.

Using YAML Config File​

To set up TLS mutual authentication via the Worker Node’s instance.yml config file, you need to change some values and also add a few keys, as shown in this example:

distributed:
  mode: managed-edge
  master:
    host: <hostname>
    port: 4200
    authToken: <token>
    tls:
      disabled: false
      rejectUnauthorized: true # false if ignoring untrusted certs
      requestCert: true
      privKeyPath: /path/to/certs/worker.key
      certPath: /path/to/certs/worker.pem
      caPath: /path/to/certs/root.pem
    resiliency: none
  group: default_fleet     

Using Environment Variables​

You can set up TLS mutual authentication by configuring this environment variable, using the format shown in this example:

CRIBL_DIST_Master_URL="tls://<authToken>@leader.cribl:4200?tls.privKeyPath=/path/to/certs/worker.key&tls.certPath=/path/to/certs/workers.pem&tls.caPath=/path/to/certs/root.pem&tls.requestCert=true&tls.rejectUnauthorized=true"

Once you’ve set this variable, restart the Worker Node. You should see the Worker Node successfully reconnect to the Leader.

If the Worker Node doesn’t connect, check cribl.log on both the Worker Node and Leader for more context about the problem. You should see errors related to dist leader communications.

To build your own Certificate Authority (a self-signed CA), see our blog post on How to Secure Cribl Stream Worker-to-Leader Communications.

TLS Mutual Authentication on Cribl.Cloud​

In Cribl.Cloud, you configure TLS mutual authentication separately for each Source.

This requires a CA certificate chain that can validate the client certificate used for authentication. You add your CA certificate by creating a new certificate entry in Cribl.Cloud.

1. Prepare the CA certificate chain PEM file.

2. Go to a Worker Node’s Settings > Security > Certificates and select Add Certificate.

3. Populate the Certificate field with any valid PEM-formatted content:

   -----BEGIN CERTIFICATE-----
   CERTIFICATE CONTENT
   -----END CERTIFICATE-----

   Copy

   The certificate and key are required only for UI validation and are not used otherwise.

4. Populate the Private key with the key in PEM format:

   -----BEGIN RSA PRIVATE KEY-----
   HIDDEN PRIVATE KEY
   -----END RSA PRIVATE KEY-----

   Copy

5. In the CA certificate field, enter your PEM-formatted certificate and save. This will generate the CA certificate path.

6. Edit the certificate again to view the certificate path and copy it.

7. Go to the Source where you want to enable mutual TLS and paste the path in the CA certificate path field.

8. Save, commit, and deploy to finish the process.

Setting Authentication on Sources/Destinations​

You can use certificates to authenticate Cribl Stream to external data senders and receivers. You configure this at the Group level, as follows:

1. Select a Group.

As an alternative to the preconfiguration in steps 2–6, you can import a certificate on the fly, using the Source’s or Destination’s Create button. See this section’s final steps below.

2. Open Group Settings (top right) > Security > Certificates.
3. Select New Certificates.
4. Paste or upload .pem files, as in Setting Up the Encrypted Channel.
5. Supply a Passphrase and/or CA certificate, if required by your integration partner’s certificate.
6. Click Save.
7. Open your relevant Source’s or Destination’s config modal.
8. Select TLS Settings (Client Side) or TLS Settings (Server Side), depending on the integration.
9. Slide Enabled on.
10. In the Certificate name drop-down, select a cert that you’ve preconfigured for this integration. This will auto-populate the corresponding fields here.
11. If you’re creating a certificate on the fly, click the Create button beside Certificate name.
12. Click Save.

Cribl Stream will create new certificates at the same Group level that you’re configuring. You can verify this at the Create new certificate modal’s bottom, by making that the Referenced table includes rows for poulated for the appropriate Sources and Destinations.

Securing the Leader Node​

This is a best practice that enables the Leader to validate itself to clients. We can secure it using the self-signed cert we created in Securing Cribl Stream:

1. Navigate to Settings > Global Settings > General Settings > API Server Settings > TLS.
2. Slide the toggle to Enabled.
3. From the Certificate Name drop-down, select a cert you’ve previously imported. This will populate the corresponding fields here.
4. Click Save.

After this save, you must prepend https:// to all Cribl Stream URLs on the Leader Node. E.g., to get back to the Settings page you just configured, you’ll now need to use https://<hostname>:<port>/settings/system)`.