Multiple Splunk Instances on Single Server

[ShaneNewman]

We have 5 16-core 2.67 GHz/48GB RAM and 3 8-core 2.39 GHz/32GB RAM Physicals. 2 of the 16 core boxes are search heads, the other 3 are indexers. 2 of the 8 core boxes are search heads (1 Job Server for summary indexing, 1 Job Server for real-time alerting), the other is an indexer.

I have noticed that none of the 16 core boxes use over 8GB RAM or 30% CPU. I have been reading several posts about an "unsupported" methodology for installing 2 instances of Splunk on a single host when that host has more than 8 cores. I have done this in our QA environment and seen no evidence of negative performance impact or conflicting configs, even with them both running separate tasks concurrently.

My question... I know I can do this on the 16-core indexers, no problem. I would like to do the same thing on the search heads because the application hardware utilization with 2 instances seems to be much better, using about 60-70% CPU/14-20GB RAM during peak. The snag seems to be that only 1 instance of Splunk Web can run at a time, even when I change the service names and port for the GUI.

Has anyone else tried this or know of a way to get around this limitation?

[gkanapathy · Accepted solution]

This will be of no benefit on search heads.

On an indexer, when you dispatch a search job, a process for each search gets created and run on each indexer instance that is in your cluster. So, if you have the CPU and IO capacity on each node (which you do), running multiple instances per node will allow each search to use more of that capacity on each node, and thus complete faster. (This wouldn't be the case if you have high search concurrency and either CPU or IO is closer to being maxed out. It will run more processes, but they'll each take longer.)

However, for any given search, you will only have a single search process on the search head. It won't run any faster or use more capacity with more search head instances. (It is in fact possible to run multiple full Splunk instances, including SplunkWeb, on a single node. But it won't buy you anything.)

Are you running on Windows OS?

[ShaneNewman]

That load is beginning to cause the search queue to get backed up and we do not have additional funds for new hardware until after the first of the year. I was hoping to find a way to mitigate that problem till then by updating the F5 to distribute users to 4 instances and get a boost.

If I understand correctly, the additional instances on the indexers may help the searches to return faster and reduce queue times though?

[ShaneNewman]

We are running WS 2008R2 currently. We have 4 AIX servers, Quad core/8GB RAM that we have scheduled to be refreshed with Red Hat. We wanted to use them as is and found out the hard way that Splunk does not like AIX for anything other than UF's. The plan was to use them as heavy forwarders for parsing all of the nix data that is coming in, 1 would be an indexer for sensitive data though (internal audit).

My hope was an additional instance on the search heads would allow us the ability to have more concurrent users. Right now, between the 2 SH's, we have about 300 users running 3K searches/day.