Problem with splunk forwarder

[trile757]

We're running Splunk Ent 9.4.15. We have a number of forwarders configured. All of them are working except one, which also serves as an rsyslog.server. Firewalls are open, selinux disabled, whitelisting disabled, all the troubleshooting commands I can find don't show any errors on either the indexer of the forwarder. Yet, the directories I've configured to monitor don't show up in an index search. All this forwarder will send is default logs from $SPLUNK_HOME/var/log/*.log. Here is just one index configured in inputs.conf:

[monitor:///drive/drive2/Linux_Audit.../audit.log

disabled = false

recursive = true

sourcetype = linux_audit (verified that this is configured on the Splunk server under settings)

index = linux_audit (verified this as well)

(drive and drive2 are just made up for security reasons. It represents a 2nd hard drive on the VM for rsyslog storage). Rsyslog is a VM running on Vmware. Splunk is a physical server. The place where the logs are stored is actually /drive/drive2/Linux_Audit/2026/10/20/hostname/audit.log. Of course, the dates change every day. 

On the forwarder I can su to splunkfwd account and I'm able to cat the audit log in any hostname folder above, so it can't be permissions. getfacl shows that splunkfwd has r-x as well. 

This worked until sometime in late 2025. Our splunk admin just told me last week that it wasn't working anymore. Aside from patching both servers there hasn't been any changes to the splunk configuration.

I'm at wit's end. Can anyone offer any suggestions?

[nachimistry]

"Connection refused" on 8089 usually means the UF isn't listening on its management port. Some hardened builds set `disableDefaultPort = true`. Check with `splunk btool server list httpServer --debug`. You can still learn a lot from the search head:

First, see what the tailing processor says about that path:

`index=_internal host=<your_uf> sourcetype=splunkd (component=TailReader OR component=WatchedFile OR component=TailingProcessor) "Linux_Audit" | stats count latest(_raw) as example by component log_level`

"Adding watch" with nothing after it means it never reads the files. Messages about checksums or CRC mean it thinks it has already seen them.

Second, make sure the data isn't arriving with bad timestamps or landing in another index:

`| tstats count min(_time) as first max(_time) as last where index=* source="*Linux_Audit*" earliest=0 latest=+10y by index sourcetype | convert ctime(first) ctime(last)`

If both come back empty, the `ignoreOlderThan = 14d` suggestion is still the best bet. Years of daily folders under one recursive monitor is a lot for a UF to walk.

[trile757]

I wound up setting 'ignoreOlderThan =2d. So the forwarder is on a rhel8 rsyslog server, so logs are being written to all day. The reason I chose 2 days is bc a script is configured to bzip all the files in every log directory from 2 days ago. This saves on disk space. 

I had it working for a time based on one of the other responses. I had also gone into inputs.conf and commented out all stanzas exact for the Linux_Audit. This morning I found more than 6.6 million events in the splunk UI. WHen I saw how many, that's when I changed ignoreOlderThan to "2d". We have data going back well into 2025 on that syslog server. 

When I saw it was working this morning, I uncommented out all the other stanzas and it stopped showing events on the indexer again. So I commented them all out again, except for Linux_Audit. Now I'm getting events again in the Splunk UI. I'm starting to think Splunk just can't ingest this much data. Is that possible?

[gcusello]

Hi @trile757 ,

I suppose that 

[monitor:///drive/drive2/Linux_Audit.../audit.log

is a mistyping because the correct one is:

[monitor:///drive/drive2/Linux_Audit/.../audit.log]

Anyway, I'd check again the permissions on that files for the user that you're using to run splunk.

Ciao.

Giuseppe

[sumansah]

Since this broke without any config change and the path is date-based (/YYYY/MM/DD/hostname/audit.log), my first suspect is how big the tree has grown. With recursive monitoring, the forwarder has to keep checking every file under Linux_Audit, which by now is several years of daily folders. Two quick checks, then the likely fix.

1. Ask the forwarder what it thinks of those files

On the UF, query the tailing processor's file status (it will prompt for the UF's admin password):

curl -k -u admin https://localhost:8089/services/admin/inputstatus/TailingProcessor:FileStatus

Look for today's audit.log paths. If they aren't listed at all, the stanza isn't matching or the UF hasn't reached them yet. If they're listed with a status or error, that tells you why they're being skipped.

Also confirm what the UF actually loaded. The stanza as posted is missing its closing ]; worth checking that's only a copy/paste slip:

$SPLUNK_HOME/bin/splunk btool inputs list --debug | grep -A6 Linux_Audit

2. Check the UF's own logs for that path

The UF is already forwarding its _internal logs, so from your search head:

index=_internal host=<your_uf> source=*splunkd.log* "Linux_Audit"

3. Likely fix: stop the UF re-checking old files

ignoreOlderThan exists for this case. The inputs.conf spec recommends it "when monitoring a directory hierarchy that contains many older, unchanging files", with a suggested value of 14d. Add it to your existing stanza:

[monitor://<your existing path>]
...
ignoreOlderThan = 14d

Then restart the UF.

⚠️Files older than ignoreOlderThan go on an ignore list and aren't checked again until the UF restarts, even if they're written to later. Don't set it shorter than the longest time a file can sit idle and still get new data. It also won't backfill anything that was missed.

If the FileStatus output shows something else for the new files, post a few of those lines and we can go from there.

Docs:

• inputs.conf spec (monitor: ignoreOlderThan, recursive)
• Troubleshoot the input process (FileStatus endpoint)

S Sah| Keep Splunkinggg...

[trile757]

I get Connection Refused with this command: curl -k -u admin https://localhost:8089/services/admin/inputstatus/TailingProcessor:FileStatus

It was a typo above. The stanza in the inputs.conf file is correct. and the btool --debug command shows me it is monitoring it. I put that file in $SPLUNKHOME/etc/apps/search/local/

With this command:  index=_internal host=<your_uf> source=*splunkd.log* "Linux_Audit" I'm now seeing "Adding watch on path .../Linux_Audit.... The UF's local logs from Splunk home are indexed. Just not the Linux_Audit path.

[trile757]

forwarder version is 9.4.9