topic Re: tstats where index=_internal no results in Monitoring Splunk

Why does the tstats search "where index=_internal" returns no results?

MaverickT — Fri, 25 Mar 2022 13:41:00 GMT

I am trying to run the following tstats search on indexer cluster, recently updated to splunk 8.2.1:

| tstats count where index=_internal by host

The search returns no results, I suspect that the reason is this message in search log of the indexer:

Mixed mode is disabled, skipping search for bucket with no TSIDX data: \opt\splunkhot\_internaldb\db\hot_v1_4334

When I check the specified bucket folder, I can see the tsidx files inside.

Interesting fact is, that this issue occurs only with _internal index, same command works fine with other indexes. I have datamodel "Splunk's Internal Server Logs" enabled and accelerated.

Any suggestions where to start troubleshooting this issue?

Re: tstats where index=_internal no results

codebuilder — Wed, 28 Jul 2021 16:46:59 GMT

Why are you running the search on an indexer and not a search head? A given indexer is only going to know about what it has stored locally whereas a SH/SHC member will be able to search across the entire instance.

Another thing to check would be to verify all your nodes are forwarding their internal logs. If you have a DMC the first/easiest place to check is Forwardeers > Forwarders Deployment > Show instances forwarding internal logs.

Re: tstats where index=_internal no results

MaverickT — Thu, 29 Jul 2021 09:56:13 GMT

Thanks for your reply. I guess I wasn't clear enough.

I run search on search head, the search log is taken from search head, but also includes log from indexer. It is taken from here:

$SPLUNK_HOME/var/run/splunk/dispatch/$SEARCH_JOB_ID/remote_logs/$INDEXER.search.log

I am sure all logs from search heads, heavy and universal forwarders are forwarded to indexer tier, since normal search (eg. index=_internal | stats count by host) produces results.

Re: tstats where index=_internal no results

burwell — Wed, 28 Jul 2021 21:43:09 GMT

So tstats fails

| tstats count where index=_internal by host

but this works?

index=_internal | stats count by host

Re: tstats where index=_internal no results

MaverickT — Thu, 29 Jul 2021 09:57:58 GMT

Yes, thats exactly the behaviour. To be more precise - tstats does not fail, it just doesnt return any results. To make things even more challenging - same tstats command works on other indexes.

Re: tstats where index=_internal no results

codebuilder — Thu, 29 Jul 2021 14:07:52 GMT

Have you checked the job inspector logs for clues about what's happening?
Run your search that returns no results then go to: Job > Inspect Job > search.log

Re: tstats where index=_internal no results

splunk219783 — Wed, 04 Aug 2021 14:50:53 GMT

Any luck with this? I actually have the same issue.

Re: tstats where index=_internal no results

splunk219783 — Thu, 12 Aug 2021 13:27:11 GMT

I have a nearly identical issue. This gives me three hosts out of ~600.

| tstats count where index=_internal by host

But this search returns 600 hosts, however it takes forever to run.

index=_internal | stats count by host

Re: tstats where index=_internal no results

codebuilder — Thu, 12 Aug 2021 16:04:09 GMT

Make sure everything under $SPLUNK_HOME is owned by the Splunk user.

Using a chown -RP splunk:splunk $SPLUNK_HOME

Re: tstats where index=_internal no results

martin_mueller — Mon, 15 Nov 2021 13:50:17 GMT

Including

include_reduced_buckets=t

in your tstats parameters should work around the 8.2 _internal tstats issue.

Re: tstats where index=_internal no results

MattibergB — Tue, 11 Jan 2022 14:51:53 GMT

Thanks for the tip, i cannot find this in knows issues though.

Are there any docs that state this bug?

Re: tstats where index=_internal no results

martin_mueller — Tue, 11 Jan 2022 15:24:58 GMT

Not that I'm aware of, no.

Support may have an SPL-Number to track.

Re: tstats where index=_internal no results

gjanders — Wed, 12 Jan 2022 01:40:09 GMT

I've been advised that 8.2.5 should likely have the fix (this may change, no guarantees), but I do not have a jira number...

Re: tstats where index=_internal no results

vgrote — Fri, 25 Mar 2022 13:31:58 GMT

Sorry to say, but I just installed 8.2.5 and ran straight into this issue 😞

VGVG

Re: tstats where index=_internal no results

gjanders — Sat, 26 Mar 2022 06:57:14 GMT

Also hit the same issue in 8.2.5, logged a new case

Note that adding the option include_reduced_buckets=t works in most cases, I've found it doesn't work when combined with PREFIX