Archive

Number of events found not matching number of events displayed

Our Splunk Enterprise deployment has started returning inconsistent results, and I've been unable to track the source of the issue. In one example, Splunk reports that it found 34 results matching the search query, but the event viewer tab below only displays 9 of the results. I ran this same query >10 times (without any changes in search terms or time window) on our search head and received this inconsistent answer about 50% of the time. Here is a screenshot:
alt text

Additionally, some queries are returning with very large numbers of reported results but displaying no results at all in the event viewer. The large number of results is expected; the query is somewhat complex and broad. But this query structure has worked consistently for us for over a year, and suddenly it is producing these inexplicable results:
alt text

It is probably relevant to mention that we used to be using two independent search heads connected to a pool of ~10 indexers. We have recently moved to a new set of two search heads connected to a new pool of ~10 indexers - all of which is mirrored at another site, where the original equipment is being used as a replicated backup.

1 Solution

Champion

I believe this is related to SPL-142964, with the workaround referenced here. It should be fixed in 6.6.4.

I doubt this late answer solves anything, but wanted to put the response here so the question could be marked as answered.

View solution in original post

0 Karma

Champion

I believe this is related to SPL-142964, with the workaround referenced here. It should be fixed in 6.6.4.

I doubt this late answer solves anything, but wanted to put the response here so the question could be marked as answered.

View solution in original post

0 Karma

We implemented the workaround and did not find that it solved the issue for us. We are working on getting authorization to roll out 6.6.4, and I'll try to remember to update this post when we are able to do so.

0 Karma

SplunkTrust
SplunkTrust

Interesting.. What's the message in the Job dropdown menu say?

It says:
[myhost] Dispatch Runner: Configuration initialization for /opt/splunk/var/run/searchpeers/myhost-1507751096 took longer than expected (1960ms) when dispatching a search (search ID: remotemyhost1507751400.34954); this typically reflects underlying storage performance issues

We have also been seeing this error a lot lately, but our Splunk admin/architect team (who have now left the organization) indicated that this was related to the migration we were undergoing and that it would resolve after the migration was completed. I believe the migration is now complete, and this error continues to appear more frequently than not.

0 Karma

SplunkTrust
SplunkTrust

Let's take a look at your IOPS.. Run this command

| rest /services/server/info

Any errors in splunkd.log?

0 Karma

Sorry for my ignorance, but when I run that command, I am not sure where in the output I am seeing IOPS. I did some research and saw advise for getting IOPS measurements from the linux command iostat. Because my $SPLUNKHOME is on device dm-6, I ran:
iostat -d dm-6 5
Scrolling output looks like this:
Device: tps kBread/s kBwrtn/s kBread kBwrtn
dm-6 240.80 0.00 12054.10 0 60270
tps is ranging between very low (<1) and ~400.

There are some errors in splunkd.log, but they are unrelated. Most have to do with an incorrect parsing of our ingested /var/log/messages (a separate issue I am working with our ops team to resolve).

0 Karma