Data missing from universal forwarder on a Linux host

[ww9rivers]

I have a puzzle with a Linux host running RHEL 8.10, which is running Splunk Universal Forwarder 9.4.1, configured to forward data from local syslog files "/var/log/secure" and "/var/log/messages" to our Splunk indexers in the Splunk Cloud.

Events from /var/log/secure are found there as expected. But no events are found from /var/log/messages.

To troubleshoot, I did find these messages in the _internal index from the host:

06-02-2025 15:01:05.507 -0400 INFO WatchedFile [3811453 tailreader0] - Will begin reading at offset=13553847 for file='/var/log/messages'.
06-01-2025 03:21:02.729 -0400 INFO WatchedFile [2392 tailreader0] - File too small to check seekcrc, probably truncated. Will re-read entire file='/var/log/messages'.

So the file was read but no events found in Splunk?

[Edit 2025-06-09]

The file inputs are configured with a simple stanza in a custom TA:

[monitor:///var/log]
whitelist = (messages$|secure$)
index = os
disabled = 0

As the stanza shows, two files are forwarded: /var/log/messages and /var/log/secure.

With this search:

| tstats count where index=os host=server-name-* by host source

I get these results:

host		source			count
server-name-a	/var/log/secure		39795
server-name-b	/var/log/messages	112960
server-name-b	/var/log/secure		21938

Server a and b are a pair running the same OS, patches, applications, etc..

[Prewin27]

@ww9rivers 

The message File too small to check seekcrc, probably truncated. Will re-read entire file indicates that Splunk detected that /var/log/messages was truncated.

Would you mind checking your logrotate setup? Splunk interprets file truncation as a reduction in size and will restart reading from the start of the file.

Troubleshooting
1-Review your logrotate config or Temporarily disable log rotation for this file to verify whether it is causing the issue.
2-Add a test message to /var/log/messages and verify whether Splunk is ingesting the new entry.

Regards,
Prewin
Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a kudos/Karma. Thanks!

[livehybrid]

Hi @ww9rivers 

Given that the log reports it is starting from offset=13553847 it sounds like the file is present and it has permission to read it, so we can rule that out. The other thing to check is that it is infact still being written to (when was the last even in that file)?

You were able to see the _internal logs for the host, so that rules out connectivity issues.

I'm wondering if perhaps its ending up somewhere else in an index you arent expecting?

How have you configured the input? Is this within the Splunk Add-on for Unix and Linux, or is this a custom monitor stanza in an inputs.conf ?

If you have the Splunk Add-on for Unix and Linux but also configured your own input then there is a chance the destination index is being overwritten by the Linux TA - its worth doing a btool to check this, on the UF try:

$SPLUNK_HOME/bin/splunk btool inputs list --debug monitor

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[ww9rivers]

Thank you very much for the detailed comments.

I edited my post with some details. I did not suspect anything with regards to the monitor stanza because another host with essentially the same configuration works as expected. Where it doesn't work, I do find events from the /var/log/secure (from the same monitor stanza).

I will run a btool debugging and report back.

Thanks again!

[PickleRick]

Have you checked that there is actually something to forward from that file? Modern RHELs don't write there by default relying on journald instead.

[ww9rivers]

I have checked the file. Besides, one of the messages (Will begin reading at offset=13553847 for file='/var/log/messages') would indicate that Splunk has found the file which has contents.

The host I have problem with is actually one in a pair (A/B) of servers. I do see events in Splunk from the B server as expected. I also find events from the A server from /var/log/secure, just nothing from /var/log/messages.

[PickleRick]

First thing to debug inputs is usually, after verifying the config checking the output of

splunk list monitor

and

splunk list inputstatus

[gcusello]

Hi @ww9rivers ,

which add-on are you using to read these files, Splunk_TA_nix or a custom add-on?

anyway, you can debug your configurations using btool (https://help.splunk.com/en/splunk-enterprise/administer/troubleshoot/9.0/first-steps/use-btool-to-tr...).

Probably there's a conflict in your configurations.

Ciao.

Giuseppe

[ww9rivers]

Thank you for your reply. I edited my post with some more details.

It's a custom TA with a simple file monitor stanza. I don't think the inputs configuration is an issue.

[dural_yyz]

Check this out.

https://docs.splunk.com/Documentation/Splunk/9.0.0/Admin/Inputsconf#MONITOR: