Why does search head cluster captain configured to run ad hoc searches only still execute savedseaches?

[support0]

Hello there,

On a Search Head Cluster (6.5.3), when performing an Health Check, I have had a warning for having a high skip ratio - between 60 & 80 %.

It seemed like it only affected the SHC captain.

I found out that, in order to reduce the load on the SHC captain - which is executing savedsearches, ad hoc searches and delegating savedsearches between other peers -, it was recommended to configure the captain to run ad hoc searches only.

It is documented here :

https://docs.splunk.com/Documentation/Splunk/7.0.0/DistSearch/Adhocclustermember#Configure_the_capta...

This way, the captain only launches ad hoc searches & still delegates seavedsearches between cluster members.

I believed this would resolve our issue.

However, the skip ratio is now of 100%, still only on the captain.

It is always 100% which is weird.

To me it is more like an issue in the way internal log are being generated.

Logs are saying : this savedsearch has been skipped on this host which is the captain, reason : the max number of auto summarization has been reached ...

While it should rather be saying : this savedsearch has been skipped on this host which is the captain, reason : captain configured to run ad hoc searches only.

The thing that makes me doubt about this is the reason savedsearches are being skipped on the captain : "the max number of auto summarization has been reached"

So I am wondering if :

it is really a good practice & it's more like a logging issue

or

it is not a good practice

Note that there are no errors on the Data Models savedsearches are flagged as being skipped -> Build 100 %

Would anyone have an idea on this ?

Thanks for any feedback!

[traxxasbreaker]

The search head cluster captain will actually show searches that it fails to dispatch to the other search head cluster members as skipped, even though those searches aren't necessarily intended to execute on the captain. Based on your description and the health of your data models, It almost sounds like you have other search accelerations out there besides your data models that either shouldn't be running or are accelerated over a long time period, and the captain is showing them as skipped because it's realizing it can/should not dispatch them to the cluster.

I've seen the error about the max number of auto summarizations being reached in cases where a user who no longer has privileges to accelerate searches still has searches out there with accelerations that do try to run. Though I'd expect the error message to be different if the user doesn't have the capability for the acceleration and I'm sure there's other scenarios where those errors could occur, I'd never seen encountered them before removing those acceleration privileges on the users in question or for searches related to users who still had the capability to accelerate searches.

If you look at the Scheduler Activity:Instance dashboard in the Monitoring Console for your captain, there is a Runtime Statistics panel a little more than halfway down. I find that sorting that table by skip ratio gives some good insight into exactly which searches are consistently failing, and from there you can look further into the specifics of those searches.

[support0]

Thanks for your feedback,

The run statistics is very handy indeed and some failing report_acceleration could surely be optimized or scheduled.

But it's more like :

DMs -> all 100 % OK
Report Accel > all Complete, no pending.

So I have this high skip ratio.

It has been reduced by configuring parallel summarization.

It still is 30 %.

But only on report_acceleration

So I will try to tune "auto_summary_perc" parameter.

But I'm like, if configuring the captain to run ad hoc searches only really is a good practice to reduce load on the captain, then I do not understand the 100 % skip ratio, and I do not undersand even more the reason still being "max # auto summarization".

If it is just a logging issue, then it would save me from having to tune the SHC here and there.

[glpadilla_sol]

Hello,

Did you get an answer for this?

One of my clients is having the same issue. Only the captain is skipping all the searches they have a lot of resources in their environment. Also we set up this

captain_is_adhoc_searchhead=true

But that didn't help.

Kind Regards.