Splunk CPU Usage Spike after upgrade to 5.0.1

[rush05]

I upgraded from 4.3.1 to 5.0.1. After starting splunk, it runs fine for a few minutes but then the CPU usage increases rapidly and remains at 99%. I was forced to reinstall 4.3.1 to get functionality back.

This issue doesn't occur on 4.3.1, did something drastic change between these two versions? We have a realtime dashboard set up and quite a few (40+) scheduled searches running. There are also 20+ forwarders in the environment. I'm not sure what other details are needed to get to the root of this issue, but I'm happy to provide what I can.

Thank you!

[mcluver]

Look to your limits.conf for the answer, that's what solved the problem for us. Specifically you may need to lower your base_max_searches if you have it defined. We also used the max_searches_perc parameter for the [scheduler] stanza to help limit the amount of searches that the Search Head was issuing to the Indexer at a given time.

Here is a quick look at our settings now that are working nicely, obviously you'll want to adjust these to something that makes sense for your architecture.

[search]
multi_threaded_setup = 1
base_max_searches = 64

[metadata]
maxcount=500000

[scheduler]
max_searches_perc = 10

[the_wolverine]

Recently worked with Splunk on a similar issue regarding cpu spike after upgrade to 5.0.3. A workaround was provided but since the issue only occurred after upgrading to 5.0.3 (from 4.3.4) it should be a bug. IOW, no changes to configuration other than upgrade.

I hope that someone from Splunk support team can comment here about this bug.

[mcluver]

We're having a very similar problem, we have a large number of scheduled searches that regularly run. After the upgrade during each run our CPU usage has been through the roof, it looks like they made some HUGE changes to the code.

Splunk FAIL.

Let's hope 6.x resolved these problems.