Why is tstats count showing 0 on _internal index?

[PickleRick]

Today I've seen something strange. I was preparing a small workshop for the customer and wanted to show the performance difference between

index=_internal | stats count

and

| tstats count where index=_internal

I was completely baffled when the second search showed me (repeatedly) count of 0.

If I run the search on any other splunk instance I have access to it shows me more or less the same number for both searches (of course they can differ slightly as the _internal is dynamic so a difference of few dozen entries is perfectly understandable).

But this one showed 0 with tstats.

Anyone encountered something like that?

I didn't have time to investigate further, I hope I get some time tomorrow to look into it but I'm puzzled. To make thing more mysterious, for other indexes tstats shows proper counts. It's just the _internal index which lies that it has no events.

It's a 8.2.6 clustered (both indexer cluster and shcluster) installation.

[BLACKBEARCO]

Could also be related to https://community.splunk.com/t5/Splunk-Enterprise/what-makes-tstats-on-internal-go-wrong/m-p/572087.

Unfortunately, the resolution in the linked issue did not seem to apply to us. It seemed very specific to the _internal index.

[richgalloway]

Is _internal tsidx-reduced on that system?

[PickleRick]

I'll have to check that but I don't think so.

[isoutamo]

Maybe it’s related to this https://community.splunk.com/t5/Splunk-Search/tstats-is-not-displaying-all-expected-hosts/m-p/602539...

If I recall right I have seen some other too?

https://splunk-usergroups.slack.com/archives/C0YK8DN2H/p1649110401076109 probably another issue related to internal indexes.

r. Ismo

[PickleRick]

Indeed, seems related. Thanks!