What would intermittently cause less events to return the raw data versus the total number of events it says it matched on?

[briancronrath]

This has been happening every now and then on our instance where we will have users run a search, it says it will return some number of events, let's say 20, but only 2 will actually show below where you can look at what the events actually contain. You could run the same exact search later and it will give you all the data, or some other user can run it and it will give you all the data.

What are some possible reasons for why it would do this? Clearly the query detected the events, so why isn't it returning it in the data?

[briancronrath]

Any chance this internal log is related?

09-26-2019 16:57:19.941 ERROR Timeliner - Ignored 3 events because they were after the commit time (0).

[dmarling]

How big are these events? If they are greater than 16777216 characters then you may be running into known defect SPL-16600: https://docs.splunk.com/Documentation/Splunk/7.3.1/ReleaseNotes/KnownIssues

2019-02-05  SPL-166001  16MB+ events are not displayed on the search results, but they will be listed on the fields sidebar and in the timeline. search.log message: "SRSSerializer - max str len exceeded - probably corrupt"

Workaround:
Make sure fields are under 16777216 characters (or 16MB, usually _raw is the biggest)

OR

Revert back to the old serialization format (CSV), however, this applies to all searches, so you won't be getting the (performance) benefits of the new format.

$SPLUNK_HOME/etc/system/local/limits.conf: [search] results_serial_format=csv