Monitoring Splunk

tstats where index=_internal no results

MaverickT
Communicator

I am trying to run the following tstats search on indexer cluster, recently updated to splunk 8.2.1:

 

| tstats count where index=_internal by host

 

 

The search returns no results, I suspect that the reason is this message in search log of the indexer:

 

Mixed mode is disabled, skipping search for bucket with no TSIDX data: \opt\splunkhot\_internaldb\db\hot_v1_4334

 

 

When I check the specified bucket folder, I can see the tsidx files inside. 

Interesting fact is, that this issue occurs only with _internal index, same command works fine with other indexes. I have datamodel "Splunk's Internal Server Logs" enabled and accelerated.

Any suggestions where to start troubleshooting this issue?

0 Karma

codebuilder
Influencer

Make sure everything under $SPLUNK_HOME is owned by the Splunk user.

Using a chown -RP splunk:splunk $SPLUNK_HOME

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

splunk219783
Path Finder

Any luck with this? I actually have the same issue.

0 Karma

codebuilder
Influencer

Why are you running the search on an indexer and not a search head? A given indexer is only going to know about what it has stored locally whereas a SH/SHC member will be able to search across the entire instance.

Another thing to check would be to verify all your nodes are forwarding their internal logs. If you have a DMC the first/easiest place to check is Forwardeers > Forwarders Deployment > Show instances forwarding internal logs.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

MaverickT
Communicator

Thanks for your reply. I guess I wasn't clear enough.

I run search on search head, the search log is taken from search head, but also includes log from indexer. It is taken from here:

 

$SPLUNK_HOME/var/run/splunk/dispatch/$SEARCH_JOB_ID/remote_logs/$INDEXER.search.log

 

 

I am sure all logs from search heads, heavy and universal forwarders are forwarded to indexer tier, since normal search (eg. index=_internal | stats count by host) produces results. 

0 Karma

burwell
SplunkTrust
SplunkTrust

So tstats fails

| tstats count where index=_internal by host

 but this works?

index=_internal | stats count by host

 

0 Karma

splunk219783
Path Finder

I have a nearly identical issue.   This gives me three hosts out of ~600.

| tstats count where index=_internal by host

 

But this search returns 600 hosts, however it takes forever to run.

index=_internal | stats count by host

 

0 Karma

MaverickT
Communicator

Yes, thats exactly the behaviour.  To be more precise - tstats does not fail, it just doesnt return any results. To make things even more challenging - same tstats command works on other indexes.

0 Karma

codebuilder
Influencer

Have you checked the job inspector logs for clues about what's happening?
Run your search that returns no results then go to:  Job > Inspect Job > search.log

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!