Splunk performance issue

[jeronssk]

Especially when alot of collegues have our dashboard opened we get a lot of delayed searches, and our deployment becomes terribbly slow! We have quite a beefy machine but it still seems to eat all of it's CPU. Is there any search finetuning we can do to get a quicker deployment?

[gcusello · Accepted solution]

Hi @jeronssk,

at first, you have to monitor the performances of your infrastructure using the Monitoring Console App.

Using it you could find that your infrastructure isn't correctly designed for the requirements (especially number of users and concurrent searches.

In addition, I hint to measure the performances of your storage system because usually it is the bottleneck of each architecture: remember that Splunk requires at least 800 IOPS (better 1200) for the storage.

You can check this using tools like Bonnie++.

Anyway, you can makes different intervenes, that I hint to perform all:

1. Use storage systems more performant,
2. improve your infrastructure, adding more resources to your Indexers and Search Heads (especially CPUs but also RAM),
3. optimize your searches.

About the first point:

• using physical indexers surely is useful, better if you have servers with many quick discks (at least 15K rpm or SSD),
• if you have to use virtual Indexers, put them in different servers to use parallel computing,

About the second point:

• check if you are using dedicated resources (as requested by Splunk) on your virtual machines,
• check if you're using the correct resources configurations in terms of CPUs and RAM, and anyway improve both of them, remember that each search in Splunk takes one CPU and release it only when the search is over,
• you could use more pipelines, using in a better way the available resources, but this solution isn't efficient if you haven't performat storage,
• for this activity I hint to engage a Splunk Architect or a Splunk Professional Service, this isn't a question for the Community!

About the third point:

• check, using the Monitoring Console, how many users and scheduled searches you have,
• check, using the Monitoring Console, if you have very heavy searches and try to optimize them using accelerations or Data Models,
• check if there are too many real time searches: they are very heavy for each system.

I hope to give you some hint to approach the problem, but, as I said, this is a job for a specialist (Architects or PS).

Ciao.

Giuseppe