There seems to be a 10 to 15 minute delay in the data that is being sent from a light weight forwarder to my central splunk server. It actually appears to pick up the changes to my log files quickly as I see it send data to the server almost instantly, but I am unable to see it on search on the receiving splunk server for quite a while. The receiving server is nearly idle. The strange thing for me is that we have our splunk server setup to receive syslog directly from other devices and that data is showing up almost instantly. Any help would be greatly appreciated.
I noticed this on quite a few light forwarders I have after I moved to my own ssl certs (and away from the stock ones shipped with splunk). On these particular light forwarders it takes about 10 minutes from when the splunk lwf is started/restarted for the data to show up on the indexer. I've never actually tracked down whether the data was already at the indexer or not sent yet. Didn't give it much thought once I knew data would flow again after that 10 minute period and after the initial delay messages would then show up in "real-time", but maybe there is something more here.
That is extremely bizarre. It is possible there are timestamp/timezone/time synchronization problems? i.e., it's possible that the data is there and indexed, but with incorrect timestamps (e.g.) and so do not return in search?
Thanks so much - That was exactly the issue. Time was off on the splunk server so the results would not show up until after its time was after the time on the client sending log data.
