I have approximately sixty Splunk forwarders sending the Windows events to my central Splunk indexer. Fours of them are AD servers, which send a bunch of events (one AD alone sends 1.3G/day), of course, and the rest are desktops, which send up only a few events in comparison. Overall, I'm indexing around 4 to 5 Gigs/day total.
However, the events that are generated today are not showing up until tomorrow. In other words, I'm getting all of the events, but they are not displaying now in the Splunk GUI in real time (or even when I search for the past few hours or day, sometimes), but tomorrow when I search for today's timerange, they will all be there.
I'm wondering if maybe the indexing of the incoming events is consuming my Splunk processing (and it's priority too, which is what I would expect) and the optimizing and search processing is taking a "back seat" until there is enough processing power to complete the optimizing and that is why it shows up late, but its all there.
Can someone confirm this might be the case or provide a different theory that makes sense?
... View more