Deployment Architecture

captain SHC recommendation

efaundez
Path Finder

Good afternoon

   Is there splunk documentation where it is reported that in a SHC the servers must be certified at the hardware level?

  I ask why, what would be the disadvantage if my captain has 40 cores and 60gb RAM compared to the other search head servers that have 60 cores and 250GB ram?

  We currently have a server that acts as a captain but he only performs AD-hoc queries and the user has almost no access to this machine, therefore he has no load and is only dedicated to making the bundle when he is captain of the cluseter.

  Any information is welcome

regards

0 Karma
1 Solution

nickhills
Ultra Champion

There is no specific specification on SHC hardware requirements, other than the standard Splunk reference specifications.
https://docs.splunk.com/Documentation/Splunk/8.0.1/Capacity/Referencehardware

Arguably, the Captain does more work in the cluster more of the time as it is responsible for a number of other processes (such as artifact replication, scheduling and alert tracking)

In your question you suggest that the captain is only servicing ad-hoc searches. Presumably you have configured this:

[shclustering]
captain_is_adhoc_searchhead = true

https://docs.splunk.com/Documentation/Splunk/8.0.1/DistSearch/Adhocclustermember

This relieves some of the burden from the captain (because its doing more work anyway) of running scheduled searches, but its just as likely to be servicing ad-hoc searches for all of your users.

As my comment above implies, the only way you can be sure which host will act as the captain is to disable dynamic election, and force one host to be the static captain.
https://docs.splunk.com/Documentation/Splunk/8.0.1/DistSearch/Staticcaptain

This approach would likely not be "supported" by Splunk because of the drawbacks such an approach brings - single point of failure being most obvious. Static captains are useful for a cluster recovery/maintenance tasks, not as a normal mode of operation.

With all of the above, the normal clustering convention (https://docs.splunk.com/Documentation/Splunk/8.0.1/DistSearch/SHCsystemrequirements) is that cluster members (be they Splunk Index Peers, Windows Exchange Servers, or a cluster of hypervisors) is that the peer nodes are the same specification.

Will it work - probably
Will Splunk support it... maybe
Is it a good idea.....

One thing I don't understand from your question is this:

the user has almost no access to this machine (the captain)

How have you configured that?

If my comment helps, please give it a thumbs up!

View solution in original post

0 Karma

nickhills
Ultra Champion

There is no specific specification on SHC hardware requirements, other than the standard Splunk reference specifications.
https://docs.splunk.com/Documentation/Splunk/8.0.1/Capacity/Referencehardware

Arguably, the Captain does more work in the cluster more of the time as it is responsible for a number of other processes (such as artifact replication, scheduling and alert tracking)

In your question you suggest that the captain is only servicing ad-hoc searches. Presumably you have configured this:

[shclustering]
captain_is_adhoc_searchhead = true

https://docs.splunk.com/Documentation/Splunk/8.0.1/DistSearch/Adhocclustermember

This relieves some of the burden from the captain (because its doing more work anyway) of running scheduled searches, but its just as likely to be servicing ad-hoc searches for all of your users.

As my comment above implies, the only way you can be sure which host will act as the captain is to disable dynamic election, and force one host to be the static captain.
https://docs.splunk.com/Documentation/Splunk/8.0.1/DistSearch/Staticcaptain

This approach would likely not be "supported" by Splunk because of the drawbacks such an approach brings - single point of failure being most obvious. Static captains are useful for a cluster recovery/maintenance tasks, not as a normal mode of operation.

With all of the above, the normal clustering convention (https://docs.splunk.com/Documentation/Splunk/8.0.1/DistSearch/SHCsystemrequirements) is that cluster members (be they Splunk Index Peers, Windows Exchange Servers, or a cluster of hypervisors) is that the peer nodes are the same specification.

Will it work - probably
Will Splunk support it... maybe
Is it a good idea.....

One thing I don't understand from your question is this:

the user has almost no access to this machine (the captain)

How have you configured that?

If my comment helps, please give it a thumbs up!
0 Karma

efaundez
Path Finder

Thanks for your prompt response, with respect to the captain server we preferably leave it assigned on a server that has little ram resource, therefore the modification of the DNS relay that has the search head ips is configured and this server is no longer It is in this list, therefore it has no user load, it is only dedicated to performing ad-hoc tasks, when accessed directly.

Why I ask this question, is that a while ago, we had the visit of a splunk architect, who informed that SHC servers must be homologated at the Hardware level since they captain in case they have fewer resources (CPU), limited to the two search head servers. will that be true? because I have not found information about it.

0 Karma

nickhills
Ultra Champion

I see:
So you have an SHC with 3 members, but only two of the members are in the DNS "pool".
The server that is excluded from the DNS list is set as a Static captain, and whilst it is set to only run ad-hoc searches, no users actually ever hit it via the SHC address?

Your question:
Splunk recommends that you use homogeneous machines with identical hardware specifications for all cluster members. The reason is that the cluster captain assigns scheduled jobs to members based on their current job loads. When it does this, it does not have insight into the actual processing power of each member's machine. Instead, it assumes that each machine is provisioned equally.

https://docs.splunk.com/Documentation/Splunk/8.0.1/DistSearch/SHCsystemrequirements

Configuring this the way you have (or are proposing) will address this issue, as the captain will only allocate scheduled searches across the two remaining members, and they will also be running all the ad-hoc jobs too.

..but.. you have introduced a single point of failure, (one of the main features of SHC) and have effectively removed 40 cores from your cluster which is 25% of search capacity.

You are also way outside of Splunk's deployment recommendations.

I totally follow the logic, but that feels like a tremendous waste of 40 cores and 60gb ram.
Not that I am recommending reducing it! 🙂

If my comment helps, please give it a thumbs up!

efaundez
Path Finder

Thank you very much, this information helps to respond to the client, informing that it is necessary to approve the SHC machines for better performance.

regards

0 Karma

nickhills
Ultra Champion

Have you set a static captain?
Obviously you can do this, but there are significant risks and drawbacks from this approach.
https://docs.splunk.com/Documentation/Splunk/8.0.1/DistSearch/Staticcaptain

If my comment helps, please give it a thumbs up!
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...