topic Re: Number of events found not matching number of events displayed in Splunk Search

Number of events found not matching number of events displayed

elliotproebstel — Thu, 12 Oct 2017 19:46:01 GMT

Our Splunk Enterprise deployment has started returning inconsistent results, and I've been unable to track the source of the issue. In one example, Splunk reports that it found 34 results matching the search query, but the event viewer tab below only displays 9 of the results. I ran this same query >10 times (without any changes in search terms or time window) on our search head and received this inconsistent answer about 50% of the time. Here is a screenshot:

Additionally, some queries are returning with very large numbers of reported results but displaying no results at all in the event viewer. The large number of results is expected; the query is somewhat complex and broad. But this query structure has worked consistently for us for over a year, and suddenly it is producing these inexplicable results:

It is probably relevant to mention that we used to be using two independent search heads connected to a pool of ~10 indexers. We have recently moved to a new set of two search heads connected to a new pool of ~10 indexers - all of which is mirrored at another site, where the original equipment is being used as a replicated backup.

Re: Number of events found not matching number of events displayed

skoelpin — Thu, 12 Oct 2017 19:56:14 GMT

Interesting.. What's the message in the Job dropdown menu say?

Re: Number of events found not matching number of events displayed

elliotproebstel — Tue, 29 Sep 2020 16:09:55 GMT

It says:
[myhost] Dispatch Runner: Configuration initialization for /opt/splunk/var/run/searchpeers/myhost-1507751096 took longer than expected (1960ms) when dispatching a search (search ID: remote_myhost_1507751400.34954); this typically reflects underlying storage performance issues

We have also been seeing this error a lot lately, but our Splunk admin/architect team (who have now left the organization) indicated that this was related to the migration we were undergoing and that it would resolve after the migration was completed. I believe the migration is now complete, and this error continues to appear more frequently than not.

Re: Number of events found not matching number of events displayed

skoelpin — Thu, 12 Oct 2017 21:03:21 GMT

Let's take a look at your IOPS.. Run this command

| rest /services/server/info

Any errors in splunkd.log?

Re: Number of events found not matching number of events displayed

elliotproebstel — Tue, 29 Sep 2020 16:10:21 GMT

Sorry for my ignorance, but when I run that command, I am not sure where in the output I am seeing IOPS. I did some research and saw advise for getting IOPS measurements from the linux command iostat. Because my $SPLUNKHOME is on device dm-6, I ran:
iostat -d dm-6 5
Scrolling output looks like this:
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
dm-6 240.80 0.00 12054.10 0 60270
tps is ranging between very low (<1) and ~400.

There are some errors in splunkd.log, but they are unrelated. Most have to do with an incorrect parsing of our ingested /var/log/messages (a separate issue I am working with our ops team to resolve).

Re: Number of events found not matching number of events displayed

micahkemp — Thu, 28 Dec 2017 19:40:28 GMT

I believe this is related to SPL-142964, with the workaround referenced here. It should be fixed in 6.6.4.

I doubt this late answer solves anything, but wanted to put the response here so the question could be marked as answered.

Re: Number of events found not matching number of events displayed

elliotproebstel — Wed, 03 Jan 2018 18:50:18 GMT

We implemented the workaround and did not find that it solved the issue for us. We are working on getting authorization to roll out 6.6.4, and I'll try to remember to update this post when we are able to do so.