topic Re: Splunk Performance Issues in Monitoring Splunk

Splunk Performance Issues

itsomana — Wed, 19 Oct 2011 13:31:01 GMT

Our Splunk server is constantly running at 98% cpu and the performance in splunk switching between different screens is terrible. I have a number of saved searches and reports which are linked to a traffic light dashboard. Is there any way I can determine is killing the splunk server?

Re: Splunk Performance Issues

JSapienza — Wed, 19 Oct 2011 18:58:00 GMT

More info would help:

What are the hardware spec's for you server ?

Should we assume that this box is serving as a search-head as well as an indexer ?

How many scheduled searches do you have ?

Have you made any edit to your limits.conf ?

Re: Splunk Performance Issues

itsomana — Tue, 25 Oct 2011 11:22:51 GMT

The splunk server is acting as a search head as well as an indexer.

The specifications of the server is as follows:

DL 380 G5 14Gb RAM 1 x Quad Intel Xeon 2Ghz processor

There is a dashboard configured which would have 36 traffic lights. Behind these traffic lights there are saved searches for each traffic light which at different intervals. The dashboard is set to refresh every 10 minutes. The total amount of scheduled searches are roughly 30.

Normally the dashboard is open up on three Pc's.If i look at the cpu on the splunk server it can be normally running is at 100%.

Re: Splunk Performance Issues

hartfoml — Tue, 25 Oct 2011 14:28:45 GMT

There is a limit.conf file that will slow down your system becasue you have too many saved searches per CPU. You could change the limit but that is risky and could cause even more issues. Personally I would work with Splunk Professional services to maximize your saved searches and dashboard to fit your hardware or increase your hardware to fit your needs. if you were to increase your hardware you would need more than 10 CPU cores. Check out the hardware planning links below for more help.

http://docs.splunk.com/Documentation/Splunk/latest/installation/SystemRequirements
http://docs.splunk.com/Documentation/Splunk/latest/Installation/CapacityplanningforalargerSplunkdeployment

CPU
Allow 1 CPU core for every 1MB/s of indexing volume
Allow 1 CPU core for Splunk's optimization routines for every 2MB/s of indexing volume
Allow 1 CPU per active searcher (be sure to account for scheduled searches)

"The Splunk server will start to queue searches if the number of concurrent searches is greater than 4 * (numberOfCores + 1)"

http://splunk-base.splunk.com/answers/82/i-keep-getting-this-max-concurrent-searches-reached-error-what-does-it-mean-and-how-do-i-fix-it

Re: Splunk Performance Issues

itsomana — Tue, 25 Oct 2011 14:57:04 GMT

hartfoml, many thanks for your reply. Could I just confirm one thing around saved searches and reports. If I have 60 saved searches and reports, however as I said 30 are scheduled to run at different intervals and the other 30 have a time range set to run at different intervals also, I assume that the latter 30 saved scheduled jobs will also impede performance?

Re: Splunk Performance Issues

hartfoml — Tue, 25 Oct 2011 15:52:41 GMT

The performance hit of "4 * (numberOfCores + 1)" is for concurrent searches but as you have three people that have a browser open and the searches running in the background I can’t say what is causing the issue. If you can monitor the Splund service either in top or in windows “perfmon” to find out if this is the cause of you high CPU use or is there another service that may be contributing.