Splunk Search

What would intermittently cause less events to return the raw data versus the total number of events it says it matched on?

briancronrath
Contributor

This has been happening every now and then on our instance where we will have users run a search, it says it will return some number of events, let's say 20, but only 2 will actually show below where you can look at what the events actually contain. You could run the same exact search later and it will give you all the data, or some other user can run it and it will give you all the data.

What are some possible reasons for why it would do this? Clearly the query detected the events, so why isn't it returning it in the data?

briancronrath
Contributor

Any chance this internal log is related?

09-26-2019 16:57:19.941 ERROR Timeliner - Ignored 3 events because they were after the commit time (0).

0 Karma

dmarling
Builder

How big are these events? If they are greater than 16777216 characters then you may be running into known defect SPL-16600: https://docs.splunk.com/Documentation/Splunk/7.3.1/ReleaseNotes/KnownIssues

2019-02-05  SPL-166001  16MB+ events are not displayed on the search results, but they will be listed on the fields sidebar and in the timeline. search.log message: "SRSSerializer - max str len exceeded - probably corrupt"

Workaround:
Make sure fields are under 16777216 characters (or 16MB, usually _raw is the biggest)

OR

Revert back to the old serialization format (CSV), however, this applies to all searches, so you won't be getting the (performance) benefits of the new format.

$SPLUNK_HOME/etc/system/local/limits.conf: [search] results_serial_format=csv
If this comment/answer was helpful, please up vote it. Thank you.
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...