I have a puzzle with a Linux host running RHEL 8.10, which is running Splunk Universal Forwarder 9.4.1, configured to forward data from local syslog files "/var/log/secure" and "/var/log/messages" to our Splunk indexers in the Splunk Cloud.
Events from /var/log/secure are found there as expected. But no events are found from /var/log/messages.
To troubleshoot, I did find these messages in the _internal index from the host:
06-02-2025 15:01:05.507 -0400 INFO WatchedFile [3811453 tailreader0] - Will begin reading at offset=13553847 for file='/var/log/messages'.
06-01-2025 03:21:02.729 -0400 INFO WatchedFile [2392 tailreader0] - File too small to check seekcrc, probably truncated. Will re-read entire file='/var/log/messages'.
[monitor:///var/log]
whitelist = (messages$|secure$)
index = os
disabled = 0
As the stanza shows, two files are forwarded: /var/log/messages and /var/log/secure.
With this search:
| tstats count where index=os host=server-name-* by host source
I get these results:
host source count
server-name-a /var/log/secure 39795
server-name-b /var/log/messages 112960
server-name-b /var/log/secure 21938
Server a and b are a pair running the same OS, patches, applications, etc..
The message File too small to check seekcrc, probably truncated. Will re-read entire file indicates that Splunk detected that /var/log/messages was truncated.
Would you mind checking your logrotate setup? Splunk interprets file truncation as a reduction in size and will restart reading from the start of the file.
Troubleshooting
1-Review your logrotate config or Temporarily disable log rotation for this file to verify whether it is causing the issue.
2-Add a test message to /var/log/messages and verify whether Splunk is ingesting the new entry.
Regards,
Prewin
Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a kudos/Karma. Thanks!
Hi @ww9rivers
Given that the log reports it is starting from offset=13553847 it sounds like the file is present and it has permission to read it, so we can rule that out. The other thing to check is that it is infact still being written to (when was the last even in that file)?
You were able to see the _internal logs for the host, so that rules out connectivity issues.
I'm wondering if perhaps its ending up somewhere else in an index you arent expecting?
How have you configured the input? Is this within the Splunk Add-on for Unix and Linux, or is this a custom monitor stanza in an inputs.conf ?
If you have the Splunk Add-on for Unix and Linux but also configured your own input then there is a chance the destination index is being overwritten by the Linux TA - its worth doing a btool to check this, on the UF try:
$SPLUNK_HOME/bin/splunk btool inputs list --debug monitor
🌟 Did this answer help you? If so, please consider:
Your feedback encourages the volunteers in this community to continue contributing
Thank you very much for the detailed comments.
I edited my post with some details. I did not suspect anything with regards to the monitor stanza because another host with essentially the same configuration works as expected. Where it doesn't work, I do find events from the /var/log/secure (from the same monitor stanza).
I will run a btool debugging and report back.
Thanks again!
Have you checked that there is actually something to forward from that file? Modern RHELs don't write there by default relying on journald instead.
I have checked the file. Besides, one of the messages (Will begin reading at offset=13553847 for file='/var/log/messages') would indicate that Splunk has found the file which has contents.
The host I have problem with is actually one in a pair (A/B) of servers. I do see events in Splunk from the B server as expected. I also find events from the A server from /var/log/secure, just nothing from /var/log/messages.
First thing to debug inputs is usually, after verifying the config checking the output of
splunk list monitor
and
splunk list inputstatus
Hi @ww9rivers ,
which add-on are you using to read these files, Splunk_TA_nix or a custom add-on?
anyway, you can debug your configurations using btool (https://help.splunk.com/en/splunk-enterprise/administer/troubleshoot/9.0/first-steps/use-btool-to-tr...).
Probably there's a conflict in your configurations.
Ciao.
Giuseppe
Thank you for your reply. I edited my post with some more details.
It's a custom TA with a simple file monitor stanza. I don't think the inputs configuration is an issue.