I am trying to run the following tstats search on indexer cluster, recently updated to splunk 8.2.1:
| tstats count where index=_internal by host
The search returns no results, I suspect that the reason is this message in search log of the indexer:
Mixed mode is disabled, skipping search for bucket with no TSIDX data: \opt\splunkhot\_internaldb\db\hot_v1_4334
When I check the specified bucket folder, I can see the tsidx files inside.
Interesting fact is, that this issue occurs only with _internal index, same command works fine with other indexes. I have datamodel "Splunk's Internal Server Logs" enabled and accelerated.
Any suggestions where to start troubleshooting this issue?
Including
include_reduced_buckets=t
in your tstats parameters should work around the 8.2 _internal tstats issue.
Thanks for the tip, i cannot find this in knows issues though.
Are there any docs that state this bug?
Not that I'm aware of, no.
Support may have an SPL-Number to track.
I've been advised that 8.2.5 should likely have the fix (this may change, no guarantees), but I do not have a jira number...
Sorry to say, but I just installed 8.2.5 and ran straight into this issue 😞
VGVG
Also hit the same issue in 8.2.5, logged a new case
Note that adding the option include_reduced_buckets=t works in most cases, I've found it doesn't work when combined with PREFIX
Make sure everything under $SPLUNK_HOME is owned by the Splunk user.
Using a chown -RP splunk:splunk $SPLUNK_HOME
Any luck with this? I actually have the same issue.
Why are you running the search on an indexer and not a search head? A given indexer is only going to know about what it has stored locally whereas a SH/SHC member will be able to search across the entire instance.
Another thing to check would be to verify all your nodes are forwarding their internal logs. If you have a DMC the first/easiest place to check is Forwardeers > Forwarders Deployment > Show instances forwarding internal logs.
Thanks for your reply. I guess I wasn't clear enough.
I run search on search head, the search log is taken from search head, but also includes log from indexer. It is taken from here:
$SPLUNK_HOME/var/run/splunk/dispatch/$SEARCH_JOB_ID/remote_logs/$INDEXER.search.log
I am sure all logs from search heads, heavy and universal forwarders are forwarded to indexer tier, since normal search (eg. index=_internal | stats count by host) produces results.
So tstats fails
| tstats count where index=_internal by host
but this works?
index=_internal | stats count by host
I have a nearly identical issue. This gives me three hosts out of ~600.
| tstats count where index=_internal by host
But this search returns 600 hosts, however it takes forever to run.
index=_internal | stats count by host
Yes, thats exactly the behaviour. To be more precise - tstats does not fail, it just doesnt return any results. To make things even more challenging - same tstats command works on other indexes.
Have you checked the job inspector logs for clues about what's happening?
Run your search that returns no results then go to: Job > Inspect Job > search.log