Log file not picked up by splunk

[tech8260]

I'm trying to get our splunk server to index the local /var/log/audit/audit.log, but no matter what I do I don't see the log entries.

I have in etc/apps/search/local/inputs.conf:
[splunktcp://9997]

[monitor:///var/log/audit]
disabled = false

However, the audit.log entries never show up in splunk searches. The same configuration works fine if I put it on our other machines, which run universal forwarders. This problem only shows up on the indexing server.

There is additionally an input configured for /var/log in the Splunk for Unix configuration, apps/unix/local/inputs.conf:
[monitor:///var/log]
disabled = false
blacklist = lastlog|rootsh/[A-Za-z0-9].[0-9].[0-9a-fA-F]$|syslog-ng/.|.([0-9]|gz)$
whitelist = (.log|log$|messages|secure|auth|mesg$|cron$|acpid$|.out|.closed)

This pickes up all files in /var/log, EXCEPT /var/log/audit/*. The same happens if I comment out the blacklist and whitelist lines.

I'm at a loss here. Anyone have any ideas?

[samhughe · Accepted solution]

Have you looked at https://wiki.splunk.com/Community:Troubleshooting_Monitor_Inputs in particular https://your-splunk-server:8089/services/admin/inputstatus/TailingProcessor:FileStatus ? Can be quite helpful in seeing why files aren't being processed.

[tech8260]

Wow, thanks, I didn't know about that (not sure how I missed that page). The inputstatus page was extremely helpful, turns out it was a permission issue as our splunk indexer is not running as root.