Can only see splunk metrics.log events and not other events from monitored files via the Splunk UF

mikeylee

Hi guys,
I’m testing cribl stream for the 1st time by getting a splunk UF to forward to a Worker on the same centos7 host (ip: 10.0.2.9).

When checking Live Data, I only see events from /opt/splunkforwarder/var/log/splunk/metrics.log
I was expecting to see events from /var/log/audit/audit.log and /var/log/secure which are listed as being monitored files:

/opt/splunkforwarder/bin/splunk list monitor

Your session is invalid. Please login.
Splunk username: admin
Password:
Monitored Directories:
$SPLUNK_HOME/var/log/splunk
/opt/splunkforwarder/var/log/splunk/audit.log
/opt/splunkforwarder/var/log/splunk/btool.log
/opt/splunkforwarder/var/log/splunk/conf.log
/opt/splunkforwarder/var/log/splunk/configuration_change.log
/opt/splunkforwarder/var/log/splunk/dfm_stderr.log
/opt/splunkforwarder/var/log/splunk/dfm_stdout.log
/opt/splunkforwarder/var/log/splunk/first_install.log
/opt/splunkforwarder/var/log/splunk/health.log
/opt/splunkforwarder/var/log/splunk/license_usage.log
/opt/splunkforwarder/var/log/splunk/mongod.log
/opt/splunkforwarder/var/log/splunk/remote_searches.log
/opt/splunkforwarder/var/log/splunk/scheduler.log
/opt/splunkforwarder/var/log/splunk/search_messages.log
/opt/splunkforwarder/var/log/splunk/searchhistory.log
/opt/splunkforwarder/var/log/splunk/splunkd-utility.log
/opt/splunkforwarder/var/log/splunk/splunkd_access.log
/opt/splunkforwarder/var/log/splunk/splunkd_stderr.log
/opt/splunkforwarder/var/log/splunk/splunkd_stdout.log
/opt/splunkforwarder/var/log/splunk/splunkd_ui_access.log
/opt/splunkforwarder/var/log/splunk/wlm_monitor.log
$SPLUNK_HOME/var/log/splunk/license_usage_summary.log
/opt/splunkforwarder/var/log/splunk/license_usage_summary.log
$SPLUNK_HOME/var/log/splunk/metrics.log
/opt/splunkforwarder/var/log/splunk/metrics.log
$SPLUNK_HOME/var/log/splunk/splunk_instrumentation_cloud.log*
/opt/splunkforwarder/var/log/splunk/splunk_instrumentation_cloud.log
$SPLUNK_HOME/var/log/splunk/splunkd.log
/opt/splunkforwarder/var/log/splunk/splunkd.log
$SPLUNK_HOME/var/log/watchdog/watchdog.log*
/opt/splunkforwarder/var/log/watchdog/watchdog.log
$SPLUNK_HOME/var/run/splunk/search_telemetry/search_telemetry.json
$SPLUNK_HOME/var/spool/splunk/tracker.log
Monitored Files:
$SPLUNK_HOME/etc/splunk.version
/var/log/audit/audit.log
/var/log/secure

When the splunk UF was forwarding directly top my standlaone instance of splunk(ip: 10.0.2.9) , I could see these events in splunk.

Below is the configuration on the splunk-UF/Worker-node:

/opt/splunkforwarder/bin/splunk list forward-server

Active forwards:
10.0.2.9:9997
Configured but inactive forwards:
None

#cat /opt/splunkforwarder/etc/system/local/outputs.conf
[tcpout:cribl]
server = 10.0.2.9:9997
sendCookedData = true
negotiateProtocolLevel = 0

[tcpout]
defaultGroup = cribl

Any ideas what I might be doing wrong?

mikeylee

My bad. Ive found the events - just had to really increase the "Capture Up to N Events" and duration. We good now. Cheers

Brendan Dalpe

Hi @mikeylee, when you run your live capture, what is the filter that you are using?

mikeylee

My bad. Ive found the events - just had to really increase the "Capture Up to N Events" and duration. We good now. Cheers

Jon Rust

There is a Pack for Splunk UF internal logs that might interest you as well.