We're running Splunk Ent 9.4.15. We have a number of forwarders configured. All of them are working except one, which also serves as an rsyslog.server. Firewalls are open, selinux disabled, whitelisting disabled, all the troubleshooting commands I can find don't show any errors on either the indexer of the forwarder. Yet, the directories I've configured to monitor don't show up in an index search. All this forwarder will send is default logs from $SPLUNK_HOME/var/log/*.log. Here is just one index configured in inputs.conf: [monitor:///drive/drive2/Linux_Audit.../audit.log disabled = false recursive = true sourcetype = linux_audit (verified that this is configured on the Splunk server under settings) index = linux_audit (verified this as well) (drive and drive2 are just made up for security reasons. It represents a 2nd hard drive on the VM for rsyslog storage). Rsyslog is a VM running on Vmware. Splunk is a physical server. The place where the logs are stored is actually /drive/drive2/Linux_Audit/2026/10/20/hostname/audit.log. Of course, the dates change every day. On the forwarder I can su to splunkfwd account and I'm able to cat the audit log in any hostname folder above, so it can't be permissions. getfacl shows that splunkfwd has r-x as well. This worked until sometime in late 2025. Our splunk admin just told me last week that it wasn't working anymore. Aside from patching both servers there hasn't been any changes to the splunk configuration. I'm at wit's end. Can anyone offer any suggestions?
... View more