Today I've seen something strange. I was preparing a small workshop for the customer and wanted to show the performance difference between
index=_internal | stats count
| tstats count where index=_internal
I was completely baffled when the second search showed me (repeatedly) count of 0.
If I run the search on any other splunk instance I have access to it shows me more or less the same number for both searches (of course they can differ slightly as the _internal is dynamic so a difference of few dozen entries is perfectly understandable).
But this one showed 0 with tstats.
Anyone encountered something like that?
I didn't have time to investigate further, I hope I get some time tomorrow to look into it but I'm puzzled. To make thing more mysterious, for other indexes tstats shows proper counts. It's just the _internal index which lies that it has no events.
It's a 8.2.6 clustered (both indexer cluster and shcluster) installation.
Could also be related to https://community.splunk.com/t5/Splunk-Enterprise/what-makes-tstats-on-internal-go-wrong/m-p/572087.
Unfortunately, the resolution in the linked issue did not seem to apply to us. It seemed very specific to the _internal index.
Is _internal tsidx-reduced on that system?
I'll have to check that but I don't think so.
Maybe it’s related to this https://community.splunk.com/t5/Splunk-Search/tstats-is-not-displaying-all-expected-hosts/m-p/602539...
If I recall right I have seen some other too?
https://splunk-usergroups.slack.com/archives/C0YK8DN2H/p1649110401076109 probably another issue related to internal indexes.
Indeed, seems related. Thanks!