Why does the tstats search "where index=_internal" returns no results?

[MaverickT]

I am trying to run the following tstats search on indexer cluster, recently updated to splunk 8.2.1:

| tstats count where index=_internal by host

The search returns no results, I suspect that the reason is this message in search log of the indexer:

Mixed mode is disabled, skipping search for bucket with no TSIDX data: \opt\splunkhot\_internaldb\db\hot_v1_4334

When I check the specified bucket folder, I can see the tsidx files inside. 

Interesting fact is, that this issue occurs only with _internal index, same command works fine with other indexes. I have datamodel "Splunk's Internal Server Logs" enabled and accelerated.

Any suggestions where to start troubleshooting this issue?

[martin_mueller]

Including

include_reduced_buckets=t

in your tstats parameters should work around the 8.2 _internal tstats issue.

[MattibergB]

Thanks for the tip, i cannot find this in knows issues though.

Are there any docs that state this bug?

[martin_mueller]

Not that I'm aware of, no.

Support may have an SPL-Number to track.

[gjanders]

I've been advised that 8.2.5 should likely have the fix (this may change, no guarantees), but I do not have a jira number...

[vgrote]

Sorry to say, but I just installed 8.2.5 and ran straight into this issue 😞

VGVG

[gjanders]

Also hit the same issue in 8.2.5, logged a new case

Note that adding the option  include_reduced_buckets=t works in most cases, I've found it doesn't work when combined with PREFIX

[codebuilder]

Make sure everything under $SPLUNK_HOME is owned by the Splunk user.

Using a chown -RP splunk:splunk $SPLUNK_HOME

[splunk219783]

Any luck with this? I actually have the same issue.

[codebuilder]

Why are you running the search on an indexer and not a search head? A given indexer is only going to know about what it has stored locally whereas a SH/SHC member will be able to search across the entire instance.

Another thing to check would be to verify all your nodes are forwarding their internal logs. If you have a DMC the first/easiest place to check is Forwardeers > Forwarders Deployment > Show instances forwarding internal logs.

[MaverickT]

Thanks for your reply. I guess I wasn't clear enough.

I run search on search head, the search log is taken from search head, but also includes log from indexer. It is taken from here:

$SPLUNK_HOME/var/run/splunk/dispatch/$SEARCH_JOB_ID/remote_logs/$INDEXER.search.log

I am sure all logs from search heads, heavy and universal forwarders are forwarded to indexer tier, since normal search (eg. index=_internal | stats count by host) produces results. 

[burwell]

So tstats fails

| tstats count where index=_internal by host

 but this works?

index=_internal | stats count by host

[splunk219783]

I have a nearly identical issue.   This gives me three hosts out of ~600.

| tstats count where index=_internal by host

But this search returns 600 hosts, however it takes forever to run.

index=_internal | stats count by host

[MaverickT]

Yes, thats exactly the behaviour.  To be more precise - tstats does not fail, it just doesnt return any results. To make things even more challenging - same tstats command works on other indexes.

[codebuilder]

Have you checked the job inspector logs for clues about what's happening?
Run your search that returns no results then go to:  Job > Inspect Job > search.log