topic Re: what makes tstats on _internal go wrong? in Splunk Enterprise

what makes tstats on _internal go wrong?

MonkeyK — Fri, 22 Oct 2021 21:20:16 GMT

My teammate and I have been trying to summarize our environment to automatically build a data dictionary. Our last feature was to add a lastSeen time to use as a rudimentary data integrity check.
Recently this has stopped working on the _internal index. As in tstats max time on _internal is a week ago, even though a straight SPL search on index=_internal returns results for today or any other arbitrary slice of time I query over the last week. This suggests to me that the tsidx is messed up for _internal.

But to make matters more confusing, yesterday I was able to submit the same query and get a correct max(_time) for index=_internal.

Does anyone have an idea of what is going on with this behavior? Better yet, what I need to do to fix it?

If it matters, this is a clustered search head environment and we also have quite a few indexers

usual results

| tstats count max(_time) as lastSeen where index=_* earliest=-20d@d latest=@m by index | convert ctime(lastSeen)

index count lastSeen

_audit	999999999	10/22/2021 15:39:59
_internal	9999999	10/14/2021 20:09:35
_introspection	999999999	10/22/2021 15:39:59
_telemetry	999	10/22/2021 12:05:05

Re: what makes tstats on _internal go wrong?

gjanders — Sun, 24 Oct 2021 05:49:17 GMT

Which splunk version? I've hit an issue that appears to be a known issue with tstats and the internal index in 8.2.2...

Re: what makes tstats on _internal go wrong?

MonkeyK — Mon, 25 Oct 2021 14:10:23 GMT

I'm on 8.2.2 as well.
If the problem is really just _internal, I'm not super concerned. But it really makes me uncomfortable that there might be errors with other indexes.

Re: what makes tstats on _internal go wrong?

gjanders — Mon, 25 Oct 2021 20:26:15 GMT

It's a known issue but not currently listed on the known issues page (I have an active support case open on it).

I did ask why it's not on the known issues page, I'll ask support again for it to be listed...

Basically for tstats on the internal index you can add include_reduced_buckets=t, and that should make your results accurate.

However I found this made the search dramatically slower so it was better to not use tstats, they haven't provided a version with a bugfix yet...