Monitoring Splunk

Slow Splunk - splunkd and auditd over 100% CPU - conflict?

Motivator

I'm working on a box right now that seems to be unnecessarily slow at both searching as well as indexing from a batch folder (files have been in there for hours and haven't been deleted yet).

One odd thing I noticed is that auditd is taking 95-110% CPU, splunkd is taking 100-175% CPU, and there's another process, setroubleshootd around 90-95%.

This is an 8-3Ghz core box, so I'm not worried about the over 100% measurements from the top command, but with auditd taking so much CPU, is it perhaps conflicting with Splunk, similar to having an antivirus system on Windows monitoring Splunk's data directories?

Tags (1)
0 Karma
1 Solution

Motivator

There was a problem with auditd and setroubleshootd. I don't know what happened exactly, but I traced it to a misconfiguration with those spitting out 20MB+ logs every couple of seconds, and Splunk was trying to eat it. One of the system admins managed to fix it and unfortunately I don't know how.

I did up Splunk indexing performance by cleaning out tens of thousands of learned csv-nnnnn sourcetypes from the Learned app.

View solution in original post

0 Karma

Motivator

There was a problem with auditd and setroubleshootd. I don't know what happened exactly, but I traced it to a misconfiguration with those spitting out 20MB+ logs every couple of seconds, and Splunk was trying to eat it. One of the system admins managed to fix it and unfortunately I don't know how.

I did up Splunk indexing performance by cleaning out tens of thousands of learned csv-nnnnn sourcetypes from the Learned app.

View solution in original post

0 Karma

SplunkTrust
SplunkTrust

I asked because of setroubleshootd, which is related to selinux. You might check to see if someone has dynamically re-enabled selinux. (It's a rare chance, but still...)

0 Karma

Motivator

I doubt it, I know Splunk checks for SELINUX on startup and it hasn't complained.

0 Karma

SplunkTrust
SplunkTrust

Are you running with selinux on or off?

0 Karma

Motivator

Ouch, I guess I hadn't noticed HOW slow it was at batch input: it seems like it's doing ~5 files per minute...

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!