High cpu usage on splunk forwarder

[remy06]

Hi,

I've installed splunk and configured it as a forwarder on one of our windows DC/file server last week and has been experiencing high cpu usage as reported by our administrator..we had to disable splunk services..

I've configured it to send wineventlogs for system,security and application and has no issues with other DC/file servers with the same settings.

Any idea what could be the problem?

Could it be due to the low disk space on the server thats causing it?

[etwilegar]

Stop blaming the boxes. Splunkd is a background process. You could make the effort to add CPU throttling into your splunkd app. Database servers have internal governors, so could you. Your app is an observer, it should be like a referee and not be seen like this.

[cmaier]

Since this is still the first hit for "forwarder cpu," I'll add my situation and solution as well...

+1 for making sure you are aware of how many files you're monitoring. I was looking at 50GB+ of logs under one folder (~6k total files) and CPU was getting out of hand. Pruned some of the older stuff out (down to ~1600 files) and it's averaging about 3% now.

I have a few systems monitoring single 20GB+ files with no problem, so file size doesn't seem to be much of an issue - at least with those.

[LewisWheeler]

Agreed - Its the quantity of files and the directory structure which makes more of a difference. You can have 50GB of data in 10 files and it'll work fine, if you then have 50GB of data in 5000 files across 50 directories your going to see CPU going high.

[t9445]

Interesting sidenote: v6.4.1 Windows UF --- if at least one matching directory does not exist, the UF will peg the CPU on the windows server - as soon as you create one that will match it is fine again.

[mvierling]

Naive use of '...' can cause CPU problems. The splunkd was using 80 to 90% of the CPU on our Forwarders. After debugging the issue we found that monitor folder traversals looking for new log files is very CPU expensive. We tracked our CPU issue to the following inputs.conf stanza:

[monitor://C:\Windows...LogFiles] disabled = false sourcetype = iis crcSalt =

This replacement fixed our CPU issue:

[monitor://C:\WINDOWS\system32\LogFiles] disabled = false sourcetype = iis crcSalt =

[bman2k2k]

had the same problem on linux servers with the 6.0.1 Universal Forwarder with the input
[monitor:///var/.../messages]
disabled = 0
sourcetype = syslog
index = linevents

[LewisWheeler]

This was my issue, I had accidentally used a full tree traversal when I thought I had told Splunk to only search for a particular file under a path. Was killing CPU on my Windows box.

[jkerai · Accepted solution]

I am assuming that forwarder is running on a windows server. If so, can you please run "procdump -h splunkd.exe -c 80 -n 3 -o splunk c:\dump\splunk". This will create dump files in c:\dump\splunk that you can send to splunk support. This should help developers find the root cause. You can download procdump from http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx.

If this is happening on Linux or similar platform, you can run 'pstack <pid_of_splunkd>' couple of times when the issue is seen and send output to splunk support. you can find pid_of_splunkd by running 'splunk status'. The first pid reported is pid_of_splunkd.

[rsanders30]

Had the same issue. Would have been nice to have Splunk update the forum with the possible cause found from the dump file.

[remy06]

Hi, I'm using a normal forwarder but I've set Forwarding defaults not to store local copy of forwarded events which shouldn't take up disk space?

[jmadi]

remy06 where you ever able to resolve the high CPU issue on your DC?

[gkanapathy]

disk space usage won't affect CPU usage, but running a standard (vs light) forwarder certainly will. does the machine have a smaller/slower CPU, or does it forward more data than others?

[Joffer]

Do you use the LightForwarder or the normal Forwarder. The normal Forwarder can/will index your data and use diskspace etc. It will then take a bigger hit on your system. How much I don't know.