How to remedy UF splunk-winevtlog.exe High CPU Usage for extended periods?

[TribesmanJohn]

Hi All

The Windows Splunk UF has a process splunk-winevtlog.exe that reads the eventlog. I am seeing on a small subset of servers that this process is consuming 100% of a virtual machine CPU (vCPU) on a small subset of servers. In many cases this behaviour can go on for days/weeks until noticed (i.e. it is not burst-type usage).

Splunk UF is v8.2.1

In some cases I see this behaviour where of two servers built at the same time and identically configured, one is fine and one is using 100% CPU.

• Attempting to set a current_only=1 value in inputs.conf stanzas does not resolve the issue.
• Restart service does not resolve issue - it manifests again after a few mins
• Restart host does not resolve issue - it manifests again after a few mins

I would be keen to hear if anyone has had similar experiences and how they have remediated the issue. 🙂

Cheers!

[TribesmanJohn]

Hi Everyone Asking,

Sadly there was no resolution for this issue. It has been suggested to try upgrading to a 9.x UF, but that has been with our application integration team for a while now....

[Thundercat88]

Have Splunk offer any further information or possible changes to the .conf to managed the CPU utilisation.

Any information that Splunk has provide you which you can share would be very useful.

Thanks again in advance.

[John_Day]

Have you had any joy in resolving this?
We are starting to see it on a very small percentage of our domain controllers. Specifically on 2008R2 (yes I know its 2008R2 but needs must)

[TribesmanJohn]

Hi John,

I did not come up with a solution for my 2008 R2 servers - ultimately being EOL there was little that could be done other than the suggestion to roll back to a 7.x UF, which did not resolve the issue for me.

On my 2016 machines, the support call is ongoing - there is a workaround to include evt_resolve_ad_obj =0 in input.conf for our security event logs stanza, but that then does cause the issue that AD SIDs no longer get resolved. It has been suggested that this issue has been seen in the 8.2.x UF and to consider either rollback to 8.0.x or upgrade to 9.x.

Hope this helps

[SimAlam]

Also experiencing this on some Server 2016 servers. Had any luck resolving?

[Thundercat88]

Hi All,

We are also experiencing the same issue with one of the AWS Domain Controller, which has experience high CPU utilisation. Our support team has identified the splunk-winevtlog.exe was using 90% of the CPU when they login, it is back to 40% range which is still quite high. They have also see this on VDI instance where the splunk-winevlog.exe was also eating up all the CPU and I had to stop and start the service.

Would appreciate on what remediation or solution was implemented and if Splunk Support have been able to come back. To confirm the Splunk UF we currently have deployed is 8.2.5.

Thanks again in advance.