Monitoring Splunk

Why does the tstats search "where index=_internal" returns no results?

MaverickT
Communicator

I am trying to run the following tstats search on indexer cluster, recently updated to splunk 8.2.1:

 

 

| tstats count where index=_internal by host

 

 

 

The search returns no results, I suspect that the reason is this message in search log of the indexer:

 

 

Mixed mode is disabled, skipping search for bucket with no TSIDX data: \opt\splunkhot\_internaldb\db\hot_v1_4334

 

 

 

When I check the specified bucket folder, I can see the tsidx files inside. 

Interesting fact is, that this issue occurs only with _internal index, same command works fine with other indexes. I have datamodel "Splunk's Internal Server Logs" enabled and accelerated.

Any suggestions where to start troubleshooting this issue?

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Including

include_reduced_buckets=t

in your tstats parameters should work around the 8.2 _internal tstats issue.

MattibergB
Path Finder

Thanks for the tip, i cannot find this in knows issues though.

Are there any docs that state this bug?

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Not that I'm aware of, no.

Support may have an SPL-Number to track.

0 Karma

gjanders
SplunkTrust
SplunkTrust

I've been advised that 8.2.5 should likely have the fix (this may change, no guarantees), but I do not have a jira number...

0 Karma

vgrote
Path Finder

Sorry to say, but I just installed 8.2.5 and ran straight into this issue 😞

VGVG

0 Karma

gjanders
SplunkTrust
SplunkTrust

Also hit the same issue in 8.2.5, logged a new case

Note that adding the option  include_reduced_buckets=t works in most cases, I've found it doesn't work when combined with PREFIX

0 Karma

codebuilder
Influencer

Make sure everything under $SPLUNK_HOME is owned by the Splunk user.

Using a chown -RP splunk:splunk $SPLUNK_HOME

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

splunk219783
Path Finder

Any luck with this? I actually have the same issue.

0 Karma

codebuilder
Influencer

Why are you running the search on an indexer and not a search head? A given indexer is only going to know about what it has stored locally whereas a SH/SHC member will be able to search across the entire instance.

Another thing to check would be to verify all your nodes are forwarding their internal logs. If you have a DMC the first/easiest place to check is Forwardeers > Forwarders Deployment > Show instances forwarding internal logs.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

MaverickT
Communicator

Thanks for your reply. I guess I wasn't clear enough.

I run search on search head, the search log is taken from search head, but also includes log from indexer. It is taken from here:

 

$SPLUNK_HOME/var/run/splunk/dispatch/$SEARCH_JOB_ID/remote_logs/$INDEXER.search.log

 

 

I am sure all logs from search heads, heavy and universal forwarders are forwarded to indexer tier, since normal search (eg. index=_internal | stats count by host) produces results. 

0 Karma

burwell
SplunkTrust
SplunkTrust

So tstats fails

| tstats count where index=_internal by host

 but this works?

index=_internal | stats count by host

 

0 Karma

splunk219783
Path Finder

I have a nearly identical issue.   This gives me three hosts out of ~600.

| tstats count where index=_internal by host

 

But this search returns 600 hosts, however it takes forever to run.

index=_internal | stats count by host

 

0 Karma

MaverickT
Communicator

Yes, thats exactly the behaviour.  To be more precise - tstats does not fail, it just doesnt return any results. To make things even more challenging - same tstats command works on other indexes.

0 Karma

codebuilder
Influencer

Have you checked the job inspector logs for clues about what's happening?
Run your search that returns no results then go to:  Job > Inspect Job > search.log

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...