I have periodically seen issues where log entries sometimes take a while longer than expected to show up on our indexers (rarely it may take 2-3 hours in some cases.) This morning I observed where some specific logs being monitored had no events forwarded in over 24 hours.
I have a saved search that runs every morning, and when reviewing the results this morning I only saw data for 3 out of our 4 Domain Controllers. Looking into why data was missing from the 4th DC, I ran the search host=hostname earliest=-24h which showed a lot of events BUT only from two different sources (One of the five Windows Event Logs I have configured and a text-based DNS log.)
I checked splukd.log, and did not find anything relevant in troublsehooting this issue. I know of using https://127.0.0.1:8089/services/admin/inputstatus/TailingProcessor:FileStatus to check on the status of monitored files, but do not know of an equivalent to check on monitored Windows Event Logs.
After restarting the Splunk Universal Forwarder, I started to see events on the indexers from most (but not all) Windows Event Logs configured. I'm thinking that since this server has so many events in these logs to process, it is just getting "stuck" trying to keep up with some of the sources.
Does anyone have any helpful techniques to troubleshoot this issue, or possible ways to configure the Universal Forwarder to better keep up with all logs?
