Why am I getting inconsistent stats?

[fredclown]

I am befuddled why the below two searches return different counts for the same period of time. The tstats one returns a smaller count. I would expect them to be the same number with tstats just finishing faster. Anyone have thoughts on this?

| tstats count where index=* index!=_*

and

index=* index!=_*
| stats count

[fredclown · Accepted solution]

I think we figured out what the discrepancy was. Sourcetype differences were expected due to tstats not dealing with sourcetype renaming and stats does.

For event count Splunk will expand some events search time if the events is multikv. It is indexed as one event so tstats will only see it that way, but it can be expanded search time.

[diogofgm]

Yes, props could be applied if you're using the SH to onboard data. But Splunk knows what should be applied in index time or search time.

Check this diagram for what is being used in each step of the indexing process (not 100% updated but I get an idea):
https://community.splunk.com/t5/Getting-Data-In/Diagrams-of-how-indexing-works-in-the-Splunk-platfor...

Oneother  thing you could try is to use a "by" clause to check in which index or source type you get fewer results just to narrow down troubleshooting. Also have a look at the job inspector and check the search log to see if there are errors or warnings showing up in one type of search and no the other.

[diogofgm]

I'm currently facing something similar in a customer but its probably not related. Nevertheless, I'm just wondering are you doing the regular stats search in fast, smart or verbose mode? Also which version of splunk are you using?
In my case, im getting the same number of results in tstats and normal stats if run it in fast mode but im getting more results if I run the same normal stats in verbose mode. All of them for the same period of time. I manage to narrow down the issue in my case to a bucket not returning results in tstats and fast mode normal stats  but returning results in verbose mode. 

[fredclown]

We are on 8.1.5. That's an interesting result. I tried it in fast, smart, and verbose. For both tstats and stats I get consistent results for each method respectively. Unfortunately they are not the same number between tstats and stats.

I know for instance if you were to count sourcetype using stats vs tstats there could be difference due to sourcetype renaming happening search time. Tstats cannot see anything that would happen search time like that. I don't know if this is possible or not with a search head but is it possible to event split in props.conf on a search head? I know that some properties in props.conf only work index time. Is line_breaker one of those? Is it at all possible that maybe an event is being split search time thus the difference in counts between stats and tstats. Much hunch is no and that I am barking up the wrong tree, but just thought I would ask as I've never really tried it on a search head as it would not make much sense to split events there ... but sometimes programs allow you to do things that don't make sense.