topic Re: tstats count showing 0 on _internal index in Splunk Enterprise

Why is tstats count showing 0 on _internal index?

PickleRick — Wed, 22 Jun 2022 17:45:28 GMT

Today I've seen something strange. I was preparing a small workshop for the customer and wanted to show the performance difference between

index=_internal | stats count

and

| tstats count where index=_internal

I was completely baffled when the second search showed me (repeatedly) count of 0.

If I run the search on any other splunk instance I have access to it shows me more or less the same number for both searches (of course they can differ slightly as the _internal is dynamic so a difference of few dozen entries is perfectly understandable).

But this one showed 0 with tstats.

Anyone encountered something like that?

I didn't have time to investigate further, I hope I get some time tomorrow to look into it but I'm puzzled. To make thing more mysterious, for other indexes tstats shows proper counts. It's just the _internal index which lies that it has no events.

It's a 8.2.6 clustered (both indexer cluster and shcluster) installation.

Re: tstats count showing 0 on _internal index

isoutamo — Tue, 21 Jun 2022 18:00:19 GMT

If I recall right I have seen some other too?

https://splunk-usergroups.slack.com/archives/C0YK8DN2H/p1649110401076109 probably another issue related to internal indexes.

r. Ismo

Re: tstats count showing 0 on _internal index

PickleRick — Tue, 21 Jun 2022 18:02:08 GMT

Indeed, seems related. Thanks!

Re: tstats count showing 0 on _internal index

richgalloway — Tue, 21 Jun 2022 18:02:49 GMT

Is _internal tsidx-reduced on that system?

Re: tstats count showing 0 on _internal index

PickleRick — Tue, 21 Jun 2022 18:04:11 GMT

I'll have to check that but I don't think so.

Re: tstats count showing 0 on _internal index

BLACKBEARCO — Tue, 21 Jun 2022 19:19:18 GMT

Could also be related to https://community.splunk.com/t5/Splunk-Enterprise/what-makes-tstats-on-internal-go-wrong/m-p/572087.

Unfortunately, the resolution in the linked issue did not seem to apply to us. It seemed very specific to the _internal index.