Getting Data In

Why does indexer use only one core?

hcpr
Path Finder

Hi.
We have a distributed environment with a total of two indexers.
These run on 12-core machines.
After upgrading to Splunk 4.3.3 (build 128297) the indexers have started acting rather strangely.
After running for a while searches never finish.
When I check the indexers, the splunkd process seems to be using only one CPU core, which (naturally) is pegged at 100%.

Normally a restart of the splunk processes will clear the problem for a while (anywhere from a few hour to a couple of days).
So far I have not found anything that seems relevant in the logs.

Does anyone have any suggestions?

Tags (4)

rruijgrok
Explorer

Saw the same on our Splunk server (RHEL 6).

Fixed it via a hint on this article:
http://blog.mozilla.org/it/2012/06/30/mysql-and-the-leap-second-high-cpu-and-the-fix/

  1. Stop splunk
  2. Stop NTP
  3. Set the date via # date -s "`date`"
  4. Start NTP
  5. Start Splunk

Step 3 and 4 can also be executed with the script mentioned in the article.

0 Karma

hcpr
Path Finder

Tried the suggestion above, and it looked like that worked for a couple of days.
But now the indexers are back to the same.
They seem to be using only one core, and forwarder connections are being dropped.
Which of course means that data is not being indexed.

I really have no idea what to search for in the splunk logs, all the normal gotchas seems to be ok.

0 Karma

rruijgrok
Explorer

that last line should read:

Step 2 and 3 can also be executed with the script mentioned in the article.

0 Karma
Get Updates on the Splunk Community!

Security Highlights: September 2022 Newsletter

 September 2022 The Splunk App for Fraud Analytics (SFA) is now Splunk SupportedUse your existing Splunk ...

Platform Highlights | September 2022 Newsletter

 September 2022 What’s New in 9.0 and How to UpgradeGet a walk through of what is new Splunk Enterprise 9.0 ...

Observability Highlights | September 2022 Newsletter

 September 2022 Splunk Observability SuiteAccess to "Classic" SignalFx Interface Will be Removed on Sept 30, ...