Number of returned events doesn't equal number of events displayed

[coltadkison]

During some searches the number of events that are supposed to be returned does not match the number of events that are actually displayed. In one instance the Events counter showed 13 events, but the timeline showed "No events found" and none were displayed. In other instances fewer events are displayed than the counter states that there should be.

In the search log there are errors for Timeliner like: "08-30-2017 12:58:47.035 ERROR Timeliner - Ignored 2 events because they were after the commit time (0).". If you add up the number of ignored events you get a number equaling the number of events that are missing from the timeline. There are also log entries like: "08-30-2017 12:58:38.909 WARN SearchResultCollator - Collector X produced chunk with startTime 1503348584.000000 when our cursor time was already 0.000000, time ordering has failed!" that may or may not be related.

Running the search again usually fixes the issue, but I'd like to resolve the underlying issue or be able to explain the cause to users that report the issue.

Has anyone seen this? Can you provide details as to why events are ignored?

[13yqiao]

A sort _time in search seems to mitigate the error for us, however, this does not fix the underlying issue.

[jhall0007 · Accepted solution]

I have a similar problem and received similar errors in the search.log file. Splunk support advised this was a bug and suggested applying the following configuration tweak:

• Edit $SPLUNK_HOME/etc/system/local/limits.conf on your indexers, and add the following:

[search]
search_keepalive_frequency = 60000

• Save and close the file, then restart the indexer instances

[coltadkison]

I tried this on our cluster, but it didn't seem to work.

Did you have success with it?

[jhall0007]

Unfortunately this did not appear to resolve the issue for us either.

[woodcock]

Open a support case.