Multiple Search Heads - Possible Clustering of Virtual and Physical?

[jlemoine]

What started as a plan to stand up a new/additional VM Search Head dedicated to a specific department in IT has turned into a possible first attempt at Search Head clustering.

In trying to segregate field extractions, dashboards, etc., I was going to stand up a virtual SH specifically for the use of one department at our company. Additionally, I thought that separate SH's might lesson the workload on Splunk, at least at the SH level, but the further I go learning how to implement my plan, the more I'm wondering if we'll actually be creating more workload on the Indexers.

To the questions:
1. Will two dedicated, non-clustered, Search Heads have a positive or negative impact on overall Splunk resources, mainly SH and IDX performance, and has anyone successfully implemented this layout, and do they recommend it?
2. If two stand-alone SH's is not the solution, and instead I just need to better learn how to implement roles to isolate extractions/dashboards and the like on my existing deployment, then is clustering a Virtual SH with a Physical SH acceptable? At this point the VM has lower CPU/RAM than the physical. The department it was originally meant for will likely not need as much power as the primary/physical SH, but being a VM resources can be increased.

Thanks in advance for your time/thoughts on the matter!

[starcher · Accepted solution]

Don't cluster Search Heads or Indexers together that have notable different levels of hardware. I am talking in the sense of SHC or Index Clustering.

It is not uncommon to have different search heads unrelated to each other pointing at the same indexing layer.

[jlemoine]

If I went with the stand-alone secondary SH, but down the road needed to consolidate, would that be as simple as copying the user specific configs from the secondary SH to the primary? I'm thinking it would just be the $SPLUNK_HOME/etc/users and etc/apps files, providing that there are no duplicates that would cause contention.

[starcher]

Not sure what you mean by consolidate? Do you mean create a new SHC and migrate into it? Then no not as easy and just copying it. You should review the docs to plan if that is what you mean. http://docs.splunk.com/Documentation/Splunk/7.0.2/DistSearch/Migratefromstandalonesearchheads

Copying knowledge objects between even stand alone search heads can be complicated if the user exists and has been using both. If you want to simply stomp on one with the content from the other then copying is not as hard.

[jlemoine]

Awesome, that's the information I was looking for. Thanks for your help starcher, much appreciated.