topic Re: Splunk Stats count discrepancy in All Apps and Add-ons

Splunk Stats count discrepancy

tiagofbmm — Thu, 28 Jun 2018 16:15:20 GMT

Hello

How would searching in VERBOSE mode and a strict timerange for index=foo host=bar | stats count return a much larger value than the number of events I see

Even if I search for index=foo host=bar in the same time frame I have much less events than what the count reports. What is wrong? How can Splunk count the events with a specific host but then not returning them?

Any ideas?

Thanks

P.S.:please note the attachments evidence

Re: Splunk Stats count discrepancy

sandeeprachuri — Thu, 28 Jun 2018 17:27:06 GMT

@tiagofbmm, Wow strange. Can you post a snapshot if possible and Splunk version please?

Thanks,
Sandeep

Re: Splunk Stats count discrepancy

skoelpin — Thu, 28 Jun 2018 17:31:32 GMT

By strict timerange, are you referring to non-relative time?

So when you run stats, its returning a value of 1 and when you strip off stats its returning zero events?

Re: Splunk Stats count discrepancy

tiagofbmm — Thu, 28 Jun 2018 17:38:02 GMT

I can't put screenshots but the version is 7.0. The searches I've done are exactly as I told you though

Re: Splunk Stats count discrepancy

tiagofbmm — Thu, 28 Jun 2018 17:39:58 GMT

Yes, not a relative time. Stats count is returning a count of for instance 290, but no events at all show up

Re: Splunk Stats count discrepancy

tiagofbmm — Thu, 28 Jun 2018 19:19:46 GMT

Yes it is just like that. Stats shows there are events in that index from that host but stripping the stats off, I see no events. Weirdest thing

Re: Splunk Stats count discrepancy

somesoni2 — Thu, 28 Jun 2018 19:41:53 GMT

Does this happens for this one sourcetype only? How big are your raw data for this sourcetype?

Re: Splunk Stats count discrepancy

tiagofbmm — Thu, 28 Jun 2018 19:45:41 GMT

It's happening to dbinput sources from dbconnect. Raw data is not very big, these are audit logs. Size is not uncommon

Re: Splunk Stats count discrepancy

scannon4 — Thu, 28 Jun 2018 19:52:25 GMT

Does it do this even if you choose All Time? What time frame are you choosing?

Re: Splunk Stats count discrepancy

somesoni2 — Thu, 28 Jun 2018 19:53:45 GMT

Strange Indeed. Do you get results in statistics tab with something like this?

index=foo host=bar | table _time _raw

Also, did you try running it in different browser?

Re: Splunk Stats count discrepancy

tiagofbmm — Thu, 28 Jun 2018 19:54:43 GMT

It happens for non relative time windows, such as yesterday, or a specific hour for the day.

Re: Splunk Stats count discrepancy

tiagofbmm — Thu, 28 Jun 2018 19:59:29 GMT

I didn't try to run it in a different browser. Currently running on Chrome. About tabling the raw and time fields, as it does not show anything at all by the search itself and it returns raw as default, I didn't try it. Will come back here when I have the result of that

Re: Splunk Stats count discrepancy

scannon4 — Thu, 28 Jun 2018 20:03:02 GMT

Very interesting. I use dbconnect so I decided to try it too. I see events. Wish I could see what you are seeing.

Re: Splunk Stats count discrepancy

scannon4 — Thu, 28 Jun 2018 20:03:47 GMT

If you do index=foo without the host, do you see events from other hosts?

Re: Splunk Stats count discrepancy

tiagofbmm — Thu, 28 Jun 2018 20:07:12 GMT

Will try to show exactly what I am seing now. It's mostly incredible I must say

Re: Splunk Stats count discrepancy

tiagofbmm — Thu, 28 Jun 2018 20:08:47 GMT

The issue is not really about not seing any event. It's about seing a really small fraction of the events comparing to what the count shows.

Re: Splunk Stats count discrepancy

cpetterborg — Thu, 28 Jun 2018 20:26:49 GMT

Then why does your question say:

I don't see any events in the events tab

Which is it? Do you see no events, or not enough events? And if the latter, how many are you seeing and what is the count from the stats command?

Re: Splunk Stats count discrepancy

scannon4 — Thu, 28 Jun 2018 21:02:18 GMT

I would use Job Inspector as well to look at every step the search took to make sure nothing weird is going on with an indexer or something. Just a thought.

Re: Splunk Stats count discrepancy

tiagofbmm — Thu, 28 Jun 2018 21:21:15 GMT

It is a discrepancy between the count and the events I see by searching them. If I narrow it enough it will get to the point where I see no events and the count to be a positive number. But wider ranges, I see much less events than the one the count shows

Re: Splunk Stats count discrepancy

cpetterborg — Thu, 28 Jun 2018 21:39:37 GMT

Have you filed a support case with Splunk?