what makes tstats on _internal go wrong?

[MonkeyK]

My teammate and I have been trying to summarize our environment to automatically build a data dictionary.  Our last feature was to add a lastSeen time to use as a rudimentary data integrity check.  
Recently this has stopped working on the _internal index.  As in tstats max time on _internal is a week ago, even though a straight SPL search on index=_internal returns results for today or any other arbitrary slice of time I query over the last week.  This suggests to me that the tsidx is messed up for _internal.  

But to make matters more confusing, yesterday I was able to submit the same query and get a correct max(_time) for index=_internal.  

Does anyone have an idea of what is going on with this behavior? Better yet, what I need to do to fix it?

If it matters, this is a clustered search head environment and we also have quite a few indexers

usual results

| tstats count max(_time) as lastSeen where index=_* earliest=-20d@d latest=@m by index
| convert ctime(lastSeen)

index count lastSeen

_audit	999999999	10/22/2021 15:39:59
_internal	9999999	10/14/2021 20:09:35
_introspection	999999999	10/22/2021 15:39:59
_telemetry	999	10/22/2021 12:05:05

[gjanders]

Which splunk version? I've hit an issue that appears to be a known issue with tstats and the internal index in 8.2.2...

[MonkeyK]

I'm on 8.2.2 as well.
If the problem is really just _internal, I'm not super concerned.  But it really makes me uncomfortable that there might be errors with other indexes.

[gjanders · Accepted solution]

It's a known issue but not currently listed on  the known issues page (I have an active support case open on it).

I did ask why it's not on the known issues page, I'll ask support again for it to be listed...

Basically for tstats on the internal index you can add include_reduced_buckets=t, and that should make your results accurate.

However I found this made the search dramatically slower so it was better to not use tstats, they haven't provided a version with a bugfix yet...