Splunk Stats count discrepancy

[tiagofbmm]

Hello

How would searching in VERBOSE mode and a strict timerange for index=foo host=bar | stats count return a much larger value than the number of events I see

Even if I search for index=foo host=bar in the same time frame I have much less events than what the count reports. What is wrong? How can Splunk count the events with a specific host but then not returning them?

Any ideas?

Thanks

P.S.:please note the attachments evidence

[cpetterborg]

Then why does your question say:

I don't see any events in the events tab

Which is it? Do you see no events, or not enough events? And if the latter, how many are you seeing and what is the count from the stats command?

[tiagofbmm]

It is a discrepancy between the count and the events I see by searching them. If I narrow it enough it will get to the point where I see no events and the count to be a positive number. But wider ranges, I see much less events than the one the count shows

[cpetterborg]

Have you filed a support case with Splunk?

[tiagofbmm]

First thing to be done in the morning, just waiting for the client to supply the needed details of his license. Was checking here if I was missing something before doing so

[cpetterborg]

Good luck on the case. It will probably take a few days to get a response about whether or not it is really a bug. Hopefully it isn't a bug, or there is a work-around that you can use. You get much faster results when it ends up being "user error." I'd actually rather find out it was that than a bug.

Please reply here about what you find out from Splunk.

[scannon4]

Very interesting. I use dbconnect so I decided to try it too. I see events. Wish I could see what you are seeing.

[tiagofbmm]

Will try to show exactly what I am seing now. It's mostly incredible I must say

[skoelpin]

By strict timerange, are you referring to non-relative time?

So when you run stats, its returning a value of 1 and when you strip off stats its returning zero events?

[tiagofbmm]

Yes it is just like that. Stats shows there are events in that index from that host but stripping the stats off, I see no events. Weirdest thing

[somesoni2]

Does this happens for this one sourcetype only? How big are your raw data for this sourcetype?

[tiagofbmm]

It's happening to dbinput sources from dbconnect. Raw data is not very big, these are audit logs. Size is not uncommon

[somesoni2]

Strange Indeed. Do you get results in statistics tab with something like this?

index=foo host=bar | table _time _raw

Also, did you try running it in different browser?

[tiagofbmm]

It happens only if I filter the results with a metadata field, such as source or host or sourcetype. If I just place index=foo | stats count then the result is coherent between the number of events (checking by the Events tab and the Statistics tab in Verbose Mode)

[tiagofbmm]

I didn't try to run it in a different browser. Currently running on Chrome. About tabling the raw and time fields, as it does not show anything at all by the search itself and it returns raw as default, I didn't try it. Will come back here when I have the result of that

[tiagofbmm]

Yes, not a relative time. Stats count is returning a count of for instance 290, but no events at all show up

[sandeeprachuri]

@tiagofbmm, Wow strange. Can you post a snapshot if possible and Splunk version please?

Thanks,
Sandeep

[tiagofbmm]

I can't put screenshots but the version is 7.0. The searches I've done are exactly as I told you though

[tiagofbmm]

Version is 7.0.4. The problematic sources are from Splunk App DBConnect version 3.1.3