Moving splunk setup from standalone to a search peer and and search head setup Question

[marcusmartin]

Good Afternoon,

Some brief background

for the longest time we have been using Splunk as a Standalone Indexer and search head combined. In its infancy it was on a physical server and it worked great. Then Virtualised environments came along and it was transfered into VMware. Since then its been pretty bad, We have had all manner of "consultants" try and figure out why the searching is so slow but nobody really can figure it out.

The actual question

I have come full circle and have managed to source a brand new HP DL360 Gen10 but I wondered should i be using this for the indexing or for the search head?

[marcusmartin]

Thanks very much for your reply. 

The storage for the VMware servers is Dell Compellent. The VM servers have been provisioned with over 50% of the recommended hardware and still they are slow. 

Apologies if i am being a little dim here but which platform would you use for the indexing and which for the search head?

My initial thought is that i would use the physical server for the indexing and a VM for the searching. does that sound about right to you?

[gcusello · Accepted solution]

Hi @marcusmartin,

in my previous answer I forgot to highlight that the resources on VM-Ware must be dedicated and not shared!

Anyway, if you don't need HA, you could maintain the Indexer on the Physical server and move the Search Head on the VM.

What's the hardware reference you have on your servers?

remember that, as you can read at https://docs.splunk.com/Documentation/Splunk/9.0.5/Capacity/Referencehardware ,  the minimum requirement for Splunk Servers (low profile) without Premium Apps is:

• 12 CPUs,
• 12 GB RAM,
• 800 IOPS for the storage,

obviously the hardware requirements grow with:

• number of users,
• number of scheduled search,
• presence of Premium Apps as Enterprise Securiuty or ITSI.

If you divide Search Head and Indexer, each of them must have the above minimum hardware reference.

ciao.

Giuseppe

[marcusmartin]

Thanks. The physical server is a DL360 gen 10 - 16core , 64Gb Ram with  2.4TB of 15k SAS disks

The Virtual servers have a similar spec but they are not dedicated to splunk. which is probably the problem. 

Anyway i will setup the new physical server as the indexer. and configure a low usage VM host to act as a search head. 

Thanks very much for your assistance. 

[gcusello]

Hi @marcusmartin ,

The physical resources on the Indexer are correct by they depends on the load.

The resources on VM-Ware must be dedicated as indicated by Splunk in the hardware reference.

In addition, you could analyze the load on your servers to understand if there's some misconfiguration or if the hardware requirements are correct: you meet the minimum requirements, but they are related to the load, the concurrent users and the scheduled searches.

Probably you could find a Certified Splunk Architect to assess your infrastructure to identify eventual problems (queues, scheduled searches, storage performances, etc...),

if you are in Italy I could propose my company, but anyway, find a Splunk Architect, or ask to your company to approve a certification path for you.

See next time.

Ciao.

Giuseppe

P.S. Karma Points are appreciated by all the Contributors 😉

[isoutamo]

Hi

couple of point when you are running Splunk on VMware. 

Basically it should work ok also in VMware environment. The most important things are IOPS from VMware. It must be continuously more than 800 preferable more than 1200. And you must multiply this with Splunk nodes and enough spare IOPS for other VMware VMs too! Without enough IOPS splunk didn't work on VMware. Also you must have enough network resources between VMs. Then one of most important things are that you never should use more vCPU per splunk VM than you have in one socket - 1 or 2. It kills splunk on VMware if one vm will use cores from several sockets. And newer over allocate vCPU:s or Memory for Splunk VMs.

With those restrictions you could run Splunk on VMware quite smoothly.

r. Ismo

[gcusello]

Hi @marcusmartin,

a virtual appliance, using the same hardware reference, is slower than a physical one, infact Splunk hints to give 30% of additional resources using a virtual appliance.

In addition a physical server has usual performant disks (at least 15krpm) that are compliant with the requirement of at least 800 IOPS, is you new infrastrcuture compiant with this requirement?

What's the hardware reference you are using?

So, at first check the storage performances (using e.g. Bonnie ++) and add more resources (especially CPUs).

using VMs you could distribute searches dividing your stand-alone server in an Indexer and a Search Head,, in this way you have more resources for indexing and searching (obviously always maintaining the minimum hardware reference)  then if you need also HA, you could think to implement an Indexer Cluster, but in this case your architecture will be more complicated and requires a Splunk Architect to design it.

For your architecture, you could read at https://www.splunk.com/en_us/pdfs/tech-brief/splunk-validated-architectures.pdf 

Ciao.

Giuseppe