We are experiencing random events dropping across multiple forwarders. We have a repro of the problem as we were doing some testing yesterday on one host and literally saw it drop about 20 events. Meaning these events never made it to Splunk. Events before and after these 20 events were indexed. There was no messages in splunkd on the forwarder indicating any sort of problems with 'dropped events'.
Also, these events are very similar in structure so it wouldnt be a time stamp issue.
Can anyone advise what is the best way to troubleshoot this problem? Or offer any advise as to what could be the issue?
... View more