Follow event from source forwarder to indexer; how to troubleshoot a missing sourcetype

[muebel]

I just set up a new splunk forwarder on a linux host. One of the inputs is a monitor of the /var/log/messages file. I have a crontab entry to write some disk information to this messages file. I am unable to find these events being indexed on the indexer.

The forwarder is able to forward other events. I have a similar monitor set up to watch the /var/log/maillog file, and I find these events on the other side, being indexed. Other sorts of events are coming in. I restarted splunk on the forwarder and checked the startup events in the splunkd log. I see an entry where it says that it has begun to tail the /var/log/messages file.

Does anybody have an idea why this particular sourcetype isn't being indexed? What else can I do to follow this sourcetype onto the indexer? Is there any particular error I should look for to explain why this sourcetype isn't being indexed?

[jrodman · Accepted solution]

The first guess would be that data is not being read from the file. You can falsify this by turning on local indexing on the forwarder, or by reviewing metrics.log per_source_thruput on the forwarder. Alternatively you could inveestigate what's going on with the file input code, by searching _internal for the filename, or by enabling more debugging info: http://www.splunk.com/wiki/Community:Troubleshooting_Monitor_Inputs

Be sure that other data categories are still being sent from the forwarder, to eliminate a general communication problem.

[muebel]

There is a unique string in the events I am logging. I searched for that string across all time and was unable to find any events. There are no events being indexed from the /var/log/messages source. Other things are being logged there beyond my disk checking entry, and I cannot find these other things.

[Lowell]

How do you know you are missing an entire sourcetype and not just a single source (/var/log/messages)? Have you tried inserting a unique message string into your log file (perhaps via logger) and then searched for it across all time (just in case you have a timestamping issue)? (This should also show you if events are being timestamped with a future date, for example as well as search across source/sourcetype/host boundaries)