The below guide will walk through considerations and example configurations when running Cribl Stream on a hardened host. Typically, these configurations are needed when running Cribl on a host that has been configured in compliance with the DISA STIG. Use this guide as a template and modify the configurations as needed to fit your environment.
SELinux
See the KB article for guidance on SELinux. The most important tip for running Stream on a server with SELinux running is to ensure that the tgz file is downloaded directly to the /opt/ directory and when the tgz file is uncompressed, it is uncompressed in the /opt/ directory. Moving files on a box with SELinux installed will cause problems.
firewalld
firewalld is a zone based firewall. Within the DoD STIG, firewalld should implement a deny-all rule and should open ports by exception.
Within Stream, the leader needs to have ports 4200/tcp and 9000/tcp opened. Workers should have port 9000/tcp opened if you are allowing worker UI access and any source ports opened as well.
NOTE: The below guide assumes that firewalld is already enabled and that ports are already configured to allow communication traffic (such as SSH). The steps below are meant to configure an existing firewalld to allow ports necessary for Stream to operate.
First, we need to determine the Zone that is currently being used. Use this command to gather this information:
$ firewall-cmd --get-active-zones
mysupersecurezone
interfaces: eth0
In this example, the zone that is in use is mysupersecurezone. In some cases, you may have multiple NICs with multiple Zones. In this case, you will need to determine the correct NIC/Zone that Stream will use.
Next, we need to add the required ports for both the Leader and Worker(s).
Leader Config
sudo firewall-cmd --zone=<ZONE> --permanent --add-port=4200/tcp
sudo firewall-cmd --zone=<ZONE> --permanent --add-port=9000/tcp
Once this is complete, you can verify the status by running the command:
sudo firewall-cmd --zone=<ZONE> --permanent --list-ports
Finally, in order for the config to take effect, you need to reload the firewalld service:
sudo firewall-cmd --reload
Worker Config
Add the necessary ports based upon your source configurations:
sudo firewall-cmd --zone=<ZONE> --permanent --add-port=9000/tcp
sudo firewall-cmd --zone=<ZONE> --permanent --add-port=<CUSTOM_PORT>/tcp
Once this is complete, you can verify the status by running the command:
sudo firewall-cmd --zone=<ZONE> --permanent --list-ports
Finally, in order for the config to take effect, you need to reload the firewalld service"
sudo firewall-cmd --reload
fapolicy
fapolicy controls the execution of applications using an allow-list approach. Based on the DISA STIG for RHEL, the organization should explicitly allow or trust applications to execute with a deny-all catch all policy.
Configuring Stream with fapolicy
Since Cribl binaries are not explicitly trusted by default, we need to tell fapolicy to allow these binaries. There are two approaches to perform this allowlisting: marking files as trusted and writing custom fapolicyd rules. Marking files as trusted is preferred due to performance reasons.
Regardless of which method you choose below, you will need to create a trust file for the
git-core directory. This only needs to be done on the Leader Node.
On the Leader node, the following commands must be run after git is installed. Anytime git is updated, the update command must also be run.
(Leader Only) Add the git directory to fapolicy. This is required for the initial git init:
sudo fapolicyd-cli --file add /usr/share/git-core/ --trust-file cribl
sudo fapolicyd-cli --update
OPTION 1: Trust File
After installing Cribl but before starting it for the first time, run the following commands:
sudo fapolicyd-cli --file add /opt/cribl/bin/ --trust-file cribl
When running version 4.4.4 and above add the protobuf-libraries to the trust file:
sudo fapolicyd-cli --file add /opt/cribl/data/protobuf-libraries/ --trust-file cribl
** After executing these commands, a trust file will be created /etc/fapolicyd/trust.d/cribl that contains the path, the size, and the SHA256 hash for each trusted file. An example trust file is provided below under Additional Resources. **
Update the fapolicy trust store after applying the above configurations. This has to be done anytime that a trusted file changes. When updating git or installing Stream updates this will need to be done:
sudo fapolicyd-cli --update
When converting a Single-Instance deployment to Distributed, or when creating a new Worker Group, an additional trust will have to be added on the Leader. Substitute the Worker Group name (or default and default_fleet if converting to Distributed).
As an example, after converting to a Distributed deployment, run the following commands. (Note: After converting to Distributed the Cribl Leader will not be able to start until these commands are run):
sudo fapolicyd-cli --file add /opt/cribl/groups/default/data/protobuf-libraries/ --trust-file cribl
sudo fapolicyd-cli --file add /opt/cribl/groups/default_fleet/data/protobuf-libraries/ --trust-file cribl
sudo fapolicyd-cli --update
When adding a new group (NewGroup in this example), apply the following commands:
sudo fapolicyd-cli --file add /opt/cribl/groups/NewGroup/data/protobuf-libraries/ --trust-file cribl
sudo fapolicyd-cli --update
If you intend to in enable fapolicy after Cribl has already been deployed and running in distributed mode, then run the below command on the Leader node for each group as well as default and default_fleet:
sudo fapolicyd-cli --file add /opt/cribl/groups/<group>/data/protobuf-libraries/ --trust-file cribl
And finally, apply the configuration:
sudo fapolicyd-cli --update
Bootstapping Workers
If fapolicy is running, the bootstrapping script will not be able to automatically start the Cribl service. After running the bootstrapping script, run the following commands:
sudo fapolicyd-cli --file add /opt/cribl/bin/ --trust-file cribl
sudo fapolicyd-cli --file add /opt/cribl/data/protobuf-libraries/ --trust-file cribl
sudo fapolicyd-cli --update
Then enable Cribl to start on boot. If you used a different username than cribl in your bootstrap script, substitute that value for the -u argument.
sudo /opt/cribl/bin/cribl boot-start enable -u cribl
sudo systemctl start cribl
Upgrading Stream with fapolicy trust files
Since fapolicy tracks individual binaries and their associated SHA256 hashes, UI based upgrades will fail when running fapolicy. The error received will be something similar to:
Upgrade failed: EPERM: operation not permitted, open '/opt/cribl/unpack.zlXiIxD.tmp/cribl/bin/criblnative.node'
To run an upgrade on the leader node when using fapolicy, following the below steps. These steps are based on our documentation:
Upgrading the Leader
- Commit and deploy your desired last version. (This will be your most recent checkpoint.)
- Optionally,
git push to your configured remote repo.
- Stop Cribl Stream.
- Optional but recommended: Back up the entire
$CRIBL_HOME directory. - Optional: Check that the Worker Nodes are still functioning as expected. In the absence of the Leader Node, they should continue to work with their last deployed configurations.
- Uncompress the new Cribl Stream version on top of the old one. Make sure that the proper permissions are applied after the new version is uncompressed.
- Run the command
sudo fapolicyd-cli --update to update the fapolicy trust file - Restart Cribl Stream and log back in.
- Wait for all the Worker Nodes to report to the Leader, and ensure that they are correctly reporting the last committed configuration version.
Upgrading Workers
- Stop Cribl Stream on each Worker Node.
- Uncompress the new version on top of the old one. Make sure that the proper permissions are applied after the new version is uncompressed
- Run the command
sudo fapolicyd-cli --update to update the fapolicy trust file - Restart Cribl Stream.
(Option 2) Running Cribl Stream with Custom Rules
The second approach to allowing applications to run involves writing custom fapolicyd rules.
Note: For performance reasons, it is better to use a trust file than a custom rules file (source: Red Hat).
If you are using a default installation of fapolicy, the Cribl rules should be placed between 30-patterns.rules and 40-bad-elf.rules within the default /etc/fapolicyd/rules.d/ directory.
In this instance, we are using 35-cribl.rules. Note: the following should be used as a template. Other exceptions may need to be put in place depending upon the specific configuration.
# Allow rule for cribl
allow perm=any dir=/usr/bin : dir=/opt/cribl/bin/
allow perm=any dir=/usr/lib/systemd/ : dir=/opt/cribl/bin/
allow perm=any exe=/opt/cribl/bin/cribl : dir=/opt/cribl/
#Allow the Cribl binaries to open all files on the system (needed for /etc/ld.so.cache, /etc/resolv.conf, /usr/bin/git)
allow perm=open exe=/opt/cribl/bin/cribl : dir=/etc/
allow perm=open exe=/opt/cribl/bin/cribl : dir=/usr/
## Needed for when git tries to read the protobuf. Only needed on Leader
allow perm=open exe=/usr/bin/git : dir=/opt/cribl/
After making configuration changes, to reload the new rules run the command
fagenrules --load
Troubleshooting with --debug-deny
When running into errors and oddities when fapolicy is enabled, we can enable fapolicy in debug mode to help diagnose the file that is causing the issue.
NOTE: It is important to ensure that fapolicy is disabled before running the following command. If fapolicy is running and the following command is executed, the box will hang. Stop the service with the command:
sudo systemctl stop fapolicyd
To start fapolicy in debug mode run the following command. This will send the output of the debug to the file fapolicy.output. Using the --debug-deny flag, we are only going to capture deny events:
sudo fapolicyd --debug-deny 2> fapolicy.output &
To stop the fapolicyd debug command:
kill <pid>
If you see errors like below, this means that the Cribl binaries are not trusted. Follow the procedure above to add the Cribl Stream binaries to be trusted.
rule=13 dec=deny_audit perm=execute auid=-1 pid=1820 exe=/usr/lib/systemd/systemd : path=/opt/cribl/bin/cribl ftype=application/x-executable trust=0
rule=13 dec=deny_audit perm=execute auid=-1 pid=1827 exe=/usr/lib/systemd/systemd : path=/opt/cribl/bin/cribl ftype=application/x-executable trust=0
The follow error will occur if the git-core directory is not trusted
rule=11 dec=deny_audit perm=open auid=-1 pid=1635 exe=/usr/bin/git : path=/usr/share/git-core/templates/hooks/fsmonitor-watchman.sample ftype=text/x-perl trust=0
Common Errors and Solutions
[cribl@rhel_leader ec2-user]$ /opt/cribl/bin/cribl startbash: /opt/cribl/bin/cribl: Operation not permitted
[ec2-user@rhel_leader ~]$ sudo /opt/cribl/bin/cribl boot-start enable -u criblinternal/modules/cjs/loader.js:1144 return process.dlopen(module, path.toNamespacedPath(filename)); ^Error: /opt/cribl/bin/kerberos.node: cannot open shared object file: Operation not permitted at Object.Module._extensions..node (internal/modules/cjs/loader.js:1144:18) at Module.load (internal/modules/cjs/loader.js:950:32) at Function.Module._load (internal/modules/cjs/loader.js:790:12) at Module.require (internal/modules/cjs/loader.js:974:19) at require (internal/modules/cjs/helpers.js:93:18) at Object../src/sluice/js/input/kafka/kerberos-bindings/kerberos.ts (/opt/cribl/bin/cribl.js:14:12107088) at r (/opt/cribl/bin/cribl.js:14:107) at Object../src/sluice/js/input/kafka/kerberos-authenticator/authenticator.ts (/opt/cribl/bin/cribl.js:14:12104474) at r (/opt/cribl/bin/cribl.js:14:107) at Object../src/sluice/js/input/util.ts (/opt/cribl/bin/cribl.js:14:12133931) { code: 'ERR_DLOPEN_FAILED'}
Both of the above errors are indicative that the Cribl bin directory is not trusted. Run the following two commands to ensure that the binaries are trusted
sudo fapolicyd-cli --file add /opt/cribl/bin/ --trust-file cribl
sudo fapolicyd-cli --update
Git not found
{"time":"2022-10-09T19:41:05.772Z","cid":"api","channel":"gitmgr","level":"warn","message":"Git not found. Config versioning will not be available","error":{"message":"fatal: --local can only be used inside a git repository\n","stack":"Error: fatal: --local can only be used inside a git repository\n\n at Object.action (/opt/cribl/bin/cribl.js:14:8731127)\n at Ji.exec (/opt/cribl/bin/cribl.js:14:8731453)\n at /opt/cribl/bin/cribl.js:14:8696667\n at new Promise (<anonymous>)\n at sr.handleTaskData (/opt/cribl/bin/cribl.js:14:8696554)\n at sr.<anonymous> (/opt/cribl/bin/cribl.js:14:8696165)\n at Generator.next (<anonymous>)\n at i (/opt/cribl/bin/cribl.js:14:8680029)\n at processTicksAndRejections (internal/process/task_queues.js:95:5)"}}
fatal: not a git repository (or any of the parent directories): .git
If you receive the any of the following error related to git, this is because git was not trusted. Follow the procedure under the section Configure Stream with fapolicy to fix.
Additional Resources
This is an example of a trust file:
[ec2-user@rhel_leader ~]$ sudo cat /etc/fapolicyd/trust.d/cribl
#AUTOGENERATED FILE VERSION 2
#This file contains a list of trusted files
#
# FULL PATH SIZE SHA256
# /home/user/my-ls 157984 61a9960bf7d255a85811f4afcac51067b8f2e4c75e21cf4f2af95319d4ed1b87
/opt/cribl/bin/cribl.init.d.template 1246 30630f30db8d6fd429e534d349a3c5ea1178eb95dad7ac12729d124ace55e5c1
/opt/cribl/bin/cribl.service.template 439 864594d5a0807edcfbb1593383c760ebfb742791516e302a8f53fa3c245cafc3
/opt/cribl/bin/cribl.sh 946 de156435dbd4c566b316e70d91b08e8eb404e22468c6fdddb69dc597668075ca
/opt/cribl/bin/install-worker.sh.template 4472 aa428d738926b2aba33066d9211f4285119e9a8ccf6a81df2418703293c71c2d
/opt/cribl/bin/start.sh 373 0b22244e96b4ae9472f41a9d1cd4bc4a98cd406aabdd3b0fdddff33e9f0036bc
/opt/cribl/bin/cribl 80968808 46ac829fb9b7cd7e02e8f564d9766d15e79a2792414b4c4e31a6938d365a9a68
/opt/cribl/bin/criblnative.node 606496 84375185ade3ff64fed0bc9bcf1e03cb15d77bf9362598964e59d713a96ec4ea
/opt/cribl/bin/kerberos.node 2078160 8be303b3780b1db0417929faaea6eae9844ba54f145365e3952e9fced32ebe27
/opt/cribl/bin/parquet.node 36158248 c20e9ad5c5fd940d34665c757eb133b8ebeeed3bb5f1c65b459b928f1dcb10a7
/opt/cribl/bin/scope 30358824 af3e44ef6184f41cd4f93bb96fd26726f81d470b7d7b7c41cc3e27def8fd5cd6
/usr/share/git-core/contrib/completion/git-completion.tcsh 4801 9142f7232f10c85c5c95866835bdc6368896d9eddc68433a39593b9f1d850293
/usr/share/git-core/contrib/completion/git-prompt.sh 17778 54ff8c23867cf3921bfc0220b7953b08f7b16f90f07d7bd472cc28daa2ade91e
/usr/share/git-core/contrib/hooks/post-receive-email 22069 2a8091438f32611d5e7925f5d0cded27b21e6e9c14a207563ad92775cfcc4bec
/usr/share/git-core/contrib/hooks/pre-auto-gc-battery 1265 3953dbbd7b876bf16c2cdc2c1657e389910fe3507635cebb91c0c665f0732c08
/usr/share/git-core/contrib/hooks/multimail/CHANGES 9950 fa125b3a43d2c1b7c7078cac315e03ca7343979a19133de44395b63ae2cacfbf
/usr/share/git-core/contrib/hooks/multimail/CONTRIBUTING.rst 2532 b336d962a0fc5fa2dfdef8304b988fe3a9ae7c2f1ecca19e5246862f714907c5
/usr/share/git-core/contrib/hooks/multimail/README.Git 589 741ae7a64784b42070c71739a064458f930ba518c35c07f260f58e756f02290a
/usr/share/git-core/contrib/hooks/multimail/README.migrate-from-post-receive-email 6096 0037ec76e20c21d27860510cab0fcbf4390dc7ad82f10cdc2b128028d939b49e
/usr/share/git-core/contrib/hooks/multimail/README.rst 34571 f6b733ee43f3718ccce3a495e5f3200d1b8bb054c5f1d78bd08559763a615b77
/usr/share/git-core/contrib/hooks/multimail/doc/customizing-emails.rst 2215 99f62da073fe2a9d1964594a36d909d0400af38b5d0ad0b01b6c7d34321998c5
/usr/share/git-core/contrib/hooks/multimail/doc/gerrit.rst 1987 4193bd5142d5cbbbc7c4c83e8b92636904e1facb516617e28f3978ce7a747908
/usr/share/git-core/contrib/hooks/multimail/doc/gitolite.rst 4436 92dcc7a23ca3407c096d483edae4d6d71d764110bcffaa81ae17be53d246a951
/usr/share/git-core/contrib/hooks/multimail/doc/troubleshooting.rst 3255 3419f3db57b5f749083b88c4f930d2c67af4f98f6011e5e3726b2a8392a5a8b9
/usr/share/git-core/contrib/hooks/multimail/git_multimail 160742 60ce0fd1b0423787fb554dcf18bb90b6815d9cff656ddbef05c1977ed35a3e95
/usr/share/git-core/contrib/hooks/multimail/migrate-mailhook-config 8011 d2b7a882576d7ebc717fe4dbbd9adf00fbfee574c854c17753b506c4a4772aaf
/usr/share/git-core/contrib/hooks/multimail/post-receive.example 3577 7d7fe98bd455bd0d04602811894a4d8cd1dcdf61b7af10f70b2738ddc203ecb6
/usr/share/git-core/contrib/hooks/setgitperms.perl 6920 4e1076233ebfbdee41c86521a5e5807072515b3e944136f29d00e67d13eb925b
/usr/share/git-core/contrib/hooks/update-paranoid 11648 6f74d3ffdf0f7359844d6accbb2cc713282bcd4542c2c29f959213644ea4a21d
/usr/share/git-core/contrib/diff-highlight 7281 7cef44fbf328c7167476df4d2d4906122d412efefab5c9d8d129c5f94b0ff864
/usr/share/git-core/templates/description 73 85ab6c163d43a17ea9cf7788308bca1466f1b0a8d1cc92e26e9bf63da4062aee
/usr/share/git-core/templates/hooks/applypatch-msg.sample 478 0223497a0b8b033aa58a3a521b8629869386cf7ab0e2f101963d328aa62193f7
/usr/share/git-core/templates/hooks/commit-msg.sample 896 1f74d5e9292979b573ebd59741d46cb93ff391acdd083d340b94370753d92437
/usr/share/git-core/templates/hooks/post-update.sample 189 81765af2daef323061dcbc5e61fc16481cb74b3bac9ad8a174b186523586f6c5
/usr/share/git-core/templates/hooks/pre-applypatch.sample 424 e15c5b469ea3e0a695bea6f2c82bcf8e62821074939ddd85b77e0007ff165475
/usr/share/git-core/templates/hooks/pre-commit.sample 1643 f9af7d95eb1231ecf2eba9770fedfa8d4797a12b02d7240e98d568201251244a
/usr/share/git-core/templates/hooks/pre-merge-commit.sample 416 d3825a70337940ebbd0a5c072984e13245920cdf8898bd225c8d27a6dfc9cb53
/usr/share/git-core/templates/hooks/pre-push.sample 1374 ecce9c7e04d3f5dd9d8ada81753dd1d549a9634b26770042b58dda00217d086a
/usr/share/git-core/templates/hooks/pre-receive.sample 544 a4c3d2b9c7bb3fd8d1441c31bd4ee71a595d66b44fcf49ddb310252320169989
/usr/share/git-core/templates/hooks/push-to-checkout.sample 2783 a53d0741798b287c6dd7afa64aee473f305e65d3f49463bb9d7408ec3b12bf5f
/usr/share/git-core/templates/hooks/update.sample 3650 8d5f2fa83e103cf08b57eaa67521df9194f45cbdbcb37da52ad586097a14d106
/usr/share/git-core/templates/hooks/fsmonitor-watchman.sample 4655 f3c0228d8e827f1c5260ac59fdd92c3d425c46e54711ef713c5a54ae0a4db2b4
/usr/share/git-core/templates/hooks/pre-rebase.sample 4898 4febce867790052338076f4e66cc47efb14879d18097d1d61c8261859eaaa7b3
/usr/share/git-core/templates/hooks/prepare-commit-msg.sample 1492 e9ddcaa4189fddd25ed97fc8c789eca7b6ca16390b2392ae3276f0c8e1aa4619
/usr/share/git-core/templates/info/exclude 240 6671fe83b7a07c8932ee89164d1f2793b2318058eb8b98dc5c06ee0a5a3b0ec1
