Are you a member of the Splunk Community?
- Find Answers
- :
- Splunk Administration
- :
- Monitoring Splunk
- :
- Re: Splunk Performance Issues
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Our Splunk server is constantly running at 98% cpu and the performance in splunk switching between different screens is terrible. I have a number of saved searches and reports which are linked to a traffic light dashboard. Is there any way I can determine is killing the splunk server?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
There is a limit.conf file that will slow down your system becasue you have too many saved searches per CPU. You could change the limit but that is risky and could cause even more issues. Personally I would work with Splunk Professional services to maximize your saved searches and dashboard to fit your hardware or increase your hardware to fit your needs. if you were to increase your hardware you would need more than 10 CPU cores. Check out the hardware planning links below for more help.
http://docs.splunk.com/Documentation/Splunk/latest/installation/SystemRequirements
http://docs.splunk.com/Documentation/Splunk/latest/Installation/CapacityplanningforalargerSplunkdepl...
CPU
Allow 1 CPU core for every 1MB/s of indexing volume
Allow 1 CPU core for Splunk's optimization routines for every 2MB/s of indexing volume
Allow 1 CPU per active searcher (be sure to account for scheduled searches)
"The Splunk server will start to queue searches if the number of concurrent searches is greater than 4 * (numberOfCores + 1)"
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
There is a limit.conf file that will slow down your system becasue you have too many saved searches per CPU. You could change the limit but that is risky and could cause even more issues. Personally I would work with Splunk Professional services to maximize your saved searches and dashboard to fit your hardware or increase your hardware to fit your needs. if you were to increase your hardware you would need more than 10 CPU cores. Check out the hardware planning links below for more help.
http://docs.splunk.com/Documentation/Splunk/latest/installation/SystemRequirements
http://docs.splunk.com/Documentation/Splunk/latest/Installation/CapacityplanningforalargerSplunkdepl...
CPU
Allow 1 CPU core for every 1MB/s of indexing volume
Allow 1 CPU core for Splunk's optimization routines for every 2MB/s of indexing volume
Allow 1 CPU per active searcher (be sure to account for scheduled searches)
"The Splunk server will start to queue searches if the number of concurrent searches is greater than 4 * (numberOfCores + 1)"
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The performance hit of "4 * (numberOfCores + 1)" is for concurrent searches but as you have three people that have a browser open and the searches running in the background I can’t say what is causing the issue. If you can monitor the Splund service either in top or in windows “perfmon” to find out if this is the cause of you high CPU use or is there another service that may be contributing.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
hartfoml, many thanks for your reply. Could I just confirm one thing around saved searches and reports. If I have 60 saved searches and reports, however as I said 30 are scheduled to run at different intervals and the other 30 have a time range set to run at different intervals also, I assume that the latter 30 saved scheduled jobs will also impede performance?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The splunk server is acting as a search head as well as an indexer.
The specifications of the server is as follows:
DL 380 G5 14Gb RAM 1 x Quad Intel Xeon 2Ghz processor
There is a dashboard configured which would have 36 traffic lights. Behind these traffic lights there are saved searches for each traffic light which at different intervals. The dashboard is set to refresh every 10 minutes. The total amount of scheduled searches are roughly 30.
Normally the dashboard is open up on three Pc's.If i look at the cpu on the splunk server it can be normally running is at 100%.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
More info would help:
What are the hardware spec's for you server ?
Should we assume that this box is serving as a search-head as well as an indexer ?
How many scheduled searches do you have ?
Have you made any edit to your limits.conf ?
Leveraging Automated Threat Analysis Across the Splunk Ecosystem
Can’t Make It to Boston? Stream .conf25 and Learn with Haya Husain
Splunk Lantern’s Guide to The Most Popular .conf25 Sessions
