Are those indexed times the times of that particular event being indexed? Why the different times? (Or are they actually just "reindexed examples" and not examples of that particular event being reindexed?)
Also can you confirm that the file absolutely doesn't change in the first few hundred characters? Because my first real guess is that something rewrites a header line in that file, and thus Splunk thinks it's a new file.
Is it just a typo that inputs.conf says index=indexers, but the table of values has index=indexer?
Lastly, these appear to be heavy forwarders - why not UF? (And if I'm wrong there, no worries).
Your theory is probably right, but why? That's the question. I've seen this happen on high-load boxes, too, when there's too many files to monitor properly in one stanza. have you thought to break up the one big stanza into a lot of little ones? How many files are being tailed at any one time?
-Rich
... View more