The issue I'm having is with an index and real time reporting that uses that index. We currently use Rabbit MQ to send JSON messages to a TCP port. The rate is only about 250 messages/second. In Splunk the messages seem to take a few minutes to completely show up. For example, when running a search of the sourcetype for last 15 minutes, the most recent minute (ex. 11:50 am) might show a total of 500 events. When I run the same search a couple of minutes later, that same minute (11:50 am) has grown to 7,000. It appears the index takes a few minutes to catch up. We are trying to run real time reports, so this is causing the reports to be inaccurate.
We have run real time reports for other indexes we create the same way, so we are a little stumped on why this one doesn’t act the same. Any help would be appreciated.
... View more