Problem with splunk forwarder

[trile757]

We're running Splunk Ent 9.4.15. We have a number of forwarders configured. All of them are working except one, which also serves as an rsyslog.server. Firewalls are open, selinux disabled, whitelisting disabled, all the troubleshooting commands I can find don't show any errors on either the indexer of the forwarder. Yet, the directories I've configured to monitor don't show up in an index search. All this forwarder will send is default logs from $SPLUNK_HOME/var/log/*.log. Here is just one index configured in inputs.conf:

[monitor:///drive/drive2/Linux_Audit.../audit.log

disabled = false

recursive = true

sourcetype = linux_audit (verified that this is configured on the Splunk server under settings)

index = linux_audit (verified this as well)

(drive and drive2 are just made up for security reasons. It represents a 2nd hard drive on the VM for rsyslog storage). Rsyslog is a VM running on Vmware. Splunk is a physical server. The place where the logs are stored is actually /drive/drive2/Linux_Audit/2026/10/20/hostname/audit.log. Of course, the dates change every day. 

On the forwarder I can su to splunkfwd account and I'm able to cat the audit log in any hostname folder above, so it can't be permissions. getfacl shows that splunkfwd has r-x as well. 

This worked until sometime in late 2025. Our splunk admin just told me last week that it wasn't working anymore. Aside from patching both servers there hasn't been any changes to the splunk configuration.

I'm at wit's end. Can anyone offer any suggestions?

[gcusello]

Hi @trile757 ,

I suppose that 

[monitor:///drive/drive2/Linux_Audit.../audit.log

is a mistyping because the correct one is:

[monitor:///drive/drive2/Linux_Audit/.../audit.log]

Anyway, I'd check again the permissions on that files for the user that you're using to run splunk.

Ciao.

Giuseppe

[sumansah]

Since this broke without any config change and the path is date-based (/YYYY/MM/DD/hostname/audit.log), my first suspect is how big the tree has grown. With recursive monitoring, the forwarder has to keep checking every file under Linux_Audit, which by now is several years of daily folders. Two quick checks, then the likely fix.

1. Ask the forwarder what it thinks of those files

On the UF, query the tailing processor's file status (it will prompt for the UF's admin password):

curl -k -u admin https://localhost:8089/services/admin/inputstatus/TailingProcessor:FileStatus

Look for today's audit.log paths. If they aren't listed at all, the stanza isn't matching or the UF hasn't reached them yet. If they're listed with a status or error, that tells you why they're being skipped.

Also confirm what the UF actually loaded. The stanza as posted is missing its closing ]; worth checking that's only a copy/paste slip:

$SPLUNK_HOME/bin/splunk btool inputs list --debug | grep -A6 Linux_Audit

2. Check the UF's own logs for that path

The UF is already forwarding its _internal logs, so from your search head:

index=_internal host=<your_uf> source=*splunkd.log* "Linux_Audit"

3. Likely fix: stop the UF re-checking old files

ignoreOlderThan exists for this case. The inputs.conf spec recommends it "when monitoring a directory hierarchy that contains many older, unchanging files", with a suggested value of 14d. Add it to your existing stanza:

[monitor://<your existing path>]
...
ignoreOlderThan = 14d

Then restart the UF.

⚠️Files older than ignoreOlderThan go on an ignore list and aren't checked again until the UF restarts, even if they're written to later. Don't set it shorter than the longest time a file can sit idle and still get new data. It also won't backfill anything that was missed.

If the FileStatus output shows something else for the new files, post a few of those lines and we can go from there.

Docs:

• inputs.conf spec (monitor: ignoreOlderThan, recursive)
• Troubleshoot the input process (FileStatus endpoint)

S Sah| Keep Splunkinggg...

[trile757]

forwarder version is 9.4.9