This article covers how to read auditd logs on a Linux machine with Cribl Edge.
Prerequisites Needed:
- Root access to a Linux machine with Edge installed
- auditd running and configured on the system
- Access to the UI of the Cribl Leader node
What is auditd?
Sysadmins use audits to discover security violations and track security-relevant information on their systems. Based on preconfigured rules and properties, the audit daemon (auditd) generates log entries to record information about the events happening on the system. Administrators use this information to analyze what went wrong with the security policies and improve them further by taking additional measures.
Configure auditd
auditd comes pre-installed on most Fedora based Linux distros and can be installed via the package manager on most Debian distros. Instructions will depend on the package manager for your distro:
For Fedora based systems:
sudo dnf -y install auditd curl
OR
sudo yum -y install auditd curl
Debian based systems:
sudo apt -y install auditd curl
auditd uses a rules file for auditing and can be found at /etc/audit/auditd.rules. It comes with a default set of audit rules but can be further enhanced and customized. Generally, the best practice with custom rules is not to modify audit rules directly, but create a custom rules file in /etc/audit/rules.d/. This example will use the default out of the box rules on an Amazon Linux 2 EC2 instance.
auditd logs are written to an access controlled directory in /var/log/audit. This ensures that only explicitly allowed users are given access to these logs. By default, only the root user will have access to view auditd logs. To collect these logs with Cribl Edge, ensure the user that will be running Edge (in this case the cribl user) has access to view the auditd logs. The recommended approach is to grant read access to a restricted log admin group and add the necessary users to this group to allow read permissions. We can modify the log group in the /etc/audit/auditd.conf file. Simply comment out the the line that reads log_group = root and add a line below that reads log_group=cribl (or your chosen group for audit log access):
After making this change the audit daemon will need to be restarted. Depending on other security measures in place, a full reboot may be required. Any interruption to the audit daemon could result in the system becoming unresponsive if auditd is set to halt on audit failure.
NOTE: This is using the log_group setting in auditd.conf to manage access in lieu of chmod or setfacl. Auditd log rotation is all handled by the audit daemon. The audit daemon will rotate the log when it hits a pre-configured size or age and generate a new audit.log with the permissions configured in audit.conf. If using a setfacl or chmod, the permissions would revert every time the log rotated and edge would no longer be able to monitor the log until permissions were corrected again.
How to Read audit.log:
auditd logging is tricky because events are written over multiple lines and may not be written in sequential order. There may be several different audits happening simultaneously that are being written into the same file. As a result a standard file monitor can struggle with aggregating and breaking events properly and may be losing context and key elements of events.
Additionally, some portions of audit.log use hexadecimal to represent strings that may be hard to decipher and don’t readily provide necessary context to the events. Linux comes with a utility specifically for reading events out of audit.log: enter ausearch. Ausearch is able to aggregate and display full audit events with clearly defined event breakers to make ingestion of audit logs much easier and more complete. Ausearch by default produces output like this:
You’ll notice in the screenshot above events span multiple lines and are broken by newlines with four hyphens. This output gives more sequential logging with easy event breakers to work with.
For further information on how to read these events, see this from RedHat.
For more on ausearch, see this from RedHat.
Setting up the Edge Collector
Now that we have identified a utility to read audit.log and line break it appropriately, we can set up our input. Due to the need to leverage ausearch, we’ll have to set up an exec source. This will execute the specified command on a scheduled interval and ingest the output of the specified command.
Log into the Leader, navigate into Edge management, select your Fleet, then navigate to the Collect tab. Once there select “Add source”, and select the “Exec” tile. This will open a configuration window. To set up the collector, we will need to know our interval and our command syntax. For this example, we'll run the command every 60 seconds, and the command syntax to use is `ausearch -i --checkpoint ./check.txt`. This command runs the ausearch utility using the interpret option to convert some of the hexadecimal fields into human readable output, while setting a checkpoint file in the local directory to record the last timestamp read. The checkpoint option is critical to avoid re-ingesting the entire audit.log every minute. The checkpoint file saves several pieces of information on where to start the next ingest:
As mentioned above, these events are broken by a pretty unique event breaker, so we will need to define a custom event breaker. To do this we will navigate to the “More” tab, select “Knowledge”, then select “Event Breaker Rules” on the left hand side. Click “Add Ruleset” on the top right:
Fill in the information in the display window then click add rule on the bottom left:
The next window that displays is the event breaker. Best practice is to implement some sort of filter to ensure this rule applies to the correct events; however, since this event breaker will be used only for our ausearch outputs we will leave the filter set to "true". We will also write a regular expression to identify the timestamp prefix to make parsing more efficient:
After the event breaker is created, we will need to add it to our collector. Navigate back to the “Collect” tab, hover over the “Exec” source and select Configure. Once there, on the left pane select “Event Breakers” then select “Add ruleset” and add the auditd event breaker that was just configured:
Commit and deploy your changes and you should now be receiving complete and properly broken auditd events!
Credit: Taylor Hanson
