How to transfer reports between clustered search heads

[yossefn]

Hi, 

I have 3 clustered Search Heads, one of them is an ES Search Head. 

The ES Search Heads holds a lot of scheduled reports that causing (in my opinion) a lot of problems with "skipped searches". I want to transfer most of the scheduled reports from the ES to another Search Head in the Cluster. 

Looking at the answers, I saw a couple of them talking about transferring from a stand-alone or non-clustered search head to a clustered etc. 

In my case, all of them are clustered and I'm just looking for the best way to move them. 

Thank you.

[burwell]

You might find this post useful.

https://community.splunk.com/t5/Alerting/Splunk-Alerting-rate/m-p/447186

Basically it is the role of the captain to distribute the searches. The link above will help you identify the reason for the skipped searches. It might be that you need to add one or more heads to your cluster to run jobs. You might need to require that the captain itself run jobs. 

I'd like to encourage you to add a fourth Splunk head. It makes your life a lot easier when you need to take one head down for whatever reason.

[yossefn]

Thank you for the advice @burwell , I will have to check this with our Engineering team. 

Is that possible to create a cluster from an already running environment and to assign a captain to it?

[isoutamo]

In SHC there should be odd number of nodes due to raft protocol. Otherwise the election of captain could be a challenge. 

[burwell]

Thanks on the even/oddness of members.

In terms of what Splunk advises, I don't see any mention of having an odd number of members: https://docs.splunk.com/Documentation/Splunk/8.0.4/DistSearch/SHCarchitecture

We have two main SHC, one with 5 heads and one with 6 heads. We haven't seen many of the election issues you mention that I am aware of.  I have also administered a SHC with 4, again without issues.

We have occasionally seen times where a head will decide that another member is more worthy to be captain, presumably based on some kind of timing issues. Never quite sure.

We live with 6 heads and have hundreds of users.

[isoutamo]

When you said clustered search heads are you meaning splunk search head cluster with deployer or something else?

If SHC then it should handle this automatically. Have you a MC (monitoring console) installed? If then you should use it and check what are reasons for skipped searches (lack of resources, permissions etc.).

r. Ismo

[yossefn]

Actually I'm not sure about the Cluster, because you question about a Deployer. 

My environment is 3 Indexers, 3 Search Heads (1 ES and 2 Ad-Hoc) and another few servers.. Monitoring Console installed on one of the Ad-Hoc servers. 

When looking at the MC I can see the main reason of skipped searches: "The maximum number of concurrent auto-summarization searches on this instance has been reached", most of them from the "Splunk_SA_CIM" app. I've temporary disabled this app yesterday and this morning I can see that the Skip Ratio is changed from 84% to 7%. 

I'll probably bring back this app to be enabled, but I have to understand how should I avoid such a behavior in the future. 

Do you have any suggestions where to start from?

Many thanks.