I'm working on a box right now that seems to be unnecessarily slow at both searching as well as indexing from a batch folder (files have been in there for hours and haven't been deleted yet).
One odd thing I noticed is that auditd is taking 95-110% CPU, splunkd is taking 100-175% CPU, and there's another process, setroubleshootd around 90-95%.
This is an 8-3Ghz core box, so I'm not worried about the over 100% measurements from the top command, but with auditd taking so much CPU, is it perhaps conflicting with Splunk, similar to having an antivirus system on Windows monitoring Splunk's data directories?
Ouch, I guess I hadn't noticed HOW slow it was at batch input: it seems like it's doing ~5 files per minute...
I asked because of setroubleshootd, which is related to selinux. You might check to see if someone has dynamically re-enabled selinux. (It's a rare chance, but still...)
There was a problem with auditd and setroubleshootd. I don't know what happened exactly, but I traced it to a misconfiguration with those spitting out 20MB+ logs every couple of seconds, and Splunk was trying to eat it. One of the system admins managed to fix it and unfortunately I don't know how.
I did up Splunk indexing performance by cleaning out tens of thousands of learned csv-nnnnn sourcetypes from the Learned app.