Reporting

How to transfer reports between clustered search heads

yossefn
Path Finder

Hi, 

I have 3 clustered Search Heads, one of them is an ES Search Head. 

The ES Search Heads holds a lot of scheduled reports that causing (in my opinion) a lot of problems with "skipped searches". I want to transfer most of the scheduled reports from the ES to another Search Head in the Cluster. 

Looking at the answers, I saw a couple of them talking about transferring from a stand-alone or non-clustered search head to a clustered etc. 

In my case, all of them are clustered and I'm just looking for the best way to move them. 

 

Thank you.

Labels (2)
0 Karma

burwell
SplunkTrust
SplunkTrust

You might find this post useful.

https://community.splunk.com/t5/Alerting/Splunk-Alerting-rate/m-p/447186

Basically it is the role of the captain to distribute the searches. The link above will help you identify the reason for the skipped searches. It might be that you need to add one or more heads to your cluster to run jobs. You might need to require that the captain itself run jobs. 

I'd like to encourage you to add a fourth Splunk head. It makes your life a lot easier when you need to take one head down for whatever reason.

 

0 Karma

yossefn
Path Finder

Thank you for the advice @burwell , I will have to check this with our Engineering team. 

Is that possible to create a cluster from an already running environment and to assign a captain to it?

0 Karma

isoutamo
SplunkTrust
SplunkTrust

In SHC there should be odd number of nodes due to raft protocol. Otherwise the election of captain could be a challenge. 

0 Karma

burwell
SplunkTrust
SplunkTrust

Thanks on the even/oddness of members.

In terms of what Splunk advises, I don't see any mention of having an odd number of members: https://docs.splunk.com/Documentation/Splunk/8.0.4/DistSearch/SHCarchitecture

We have two main SHC, one with 5 heads and one with 6 heads. We haven't seen many of the election issues you mention that I am aware of.  I have also administered a SHC with 4, again without issues.

We have occasionally seen times where a head will decide that another member is more worthy to be captain, presumably based on some kind of timing issues. Never quite sure.

We live with 6 heads and have hundreds of users.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

When you said clustered search heads are you meaning splunk search head cluster with deployer or something else?

If SHC then it should handle this automatically. Have you a MC (monitoring console) installed? If then you should use it and check what are reasons for skipped searches (lack of resources, permissions etc.).

r. Ismo

0 Karma

yossefn
Path Finder

Actually I'm not sure about the Cluster, because you question about a Deployer. 

My environment is 3 Indexers, 3 Search Heads (1 ES and 2 Ad-Hoc) and another few servers.. Monitoring Console installed on one of the Ad-Hoc servers. 

When looking at the MC I can see the main reason of skipped searches: "The maximum number of concurrent auto-summarization searches on this instance has been reached", most of them from the "Splunk_SA_CIM" appI've temporary disabled this app yesterday and this morning I can see that the Skip Ratio is changed from 84% to 7%. 

I'll probably bring back this app to be enabled, but I have to understand how should I avoid such a behavior in the future. 

Do you have any suggestions where to start from?

Many thanks. 

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...