My teammate and I have been trying to summarize our environment to automatically build a data dictionary. Our last feature was to add a lastSeen time to use as a rudimentary data integrity check.
Recently this has stopped working on the _internal index. As in tstats max time on _internal is a week ago, even though a straight SPL search on index=_internal returns results for today or any other arbitrary slice of time I query over the last week. This suggests to me that the tsidx is messed up for _internal.
But to make matters more confusing, yesterday I was able to submit the same query and get a correct max(_time) for index=_internal.
Does anyone have an idea of what is going on with this behavior? Better yet, what I need to do to fix it?
If it matters, this is a clustered search head environment and we also have quite a few indexers
usual results
| tstats count max(_time) as lastSeen where index=_* earliest=-20d@d latest=@m by index
| convert ctime(lastSeen)
index count lastSeen
_audit | 999999999 | 10/22/2021 15:39:59 |
_internal | 9999999 | 10/14/2021 20:09:35 |
_introspection | 999999999 | 10/22/2021 15:39:59 |
_telemetry | 999 | 10/22/2021 12:05:05 |
It's a known issue but not currently listed on the known issues page (I have an active support case open on it).
I did ask why it's not on the known issues page, I'll ask support again for it to be listed...
Basically for tstats on the internal index you can add include_reduced_buckets=t, and that should make your results accurate.
However I found this made the search dramatically slower so it was better to not use tstats, they haven't provided a version with a bugfix yet...
Which splunk version? I've hit an issue that appears to be a known issue with tstats and the internal index in 8.2.2...
I'm on 8.2.2 as well.
If the problem is really just _internal, I'm not super concerned. But it really makes me uncomfortable that there might be errors with other indexes.
It's a known issue but not currently listed on the known issues page (I have an active support case open on it).
I did ask why it's not on the known issues page, I'll ask support again for it to be listed...
Basically for tstats on the internal index you can add include_reduced_buckets=t, and that should make your results accurate.
However I found this made the search dramatically slower so it was better to not use tstats, they haven't provided a version with a bugfix yet...