Why are these additional Splunk processes starting and stopping on Windows hosts configured with Universal Forwarders?

[TAE2112]

I have a collection of Windows 2008R2 servers running the Universal Forwarder which I configured to forward Windows Event Logs (Application and System) to a specific index on a dedicated Splunk Indexer server. Because I'm pretty new to Splunk, I'm of the opinion that it's a pretty basic setup, with nothing really fancy going on. I simply installed the Universal Forwarder and (eventually) figured out how to collect the Event logs and send them to the Indexer.

Over the past few weeks, I've been chasing an issue with CPU utilization on a few of these servers (the ones in the "collection") with little luck. The Development group has observed that the process "splunk-admon.exe" has been consuming up to 100% of the CPU on servers where the Universal Forwarders are installed. After sitting on one of the problem servers for a while, I've noticed that in addition to the persistent processes, there are a few Splunk-related processes which keep popping up and then closing:

splunkd.exe (constantly running)
splunk-winevtlog.exe (constantly running)
splunk-MonitorNoHandle.exe (start/stop)
splunk-winprintmon.exe (start/stop)
splunk-netmon.exe (start/stop)
splunk-admon.exe (start/stop)
splunk-regmon.exe (start/stop)

I expected that the splunkd.exe and splunk-winevtlog.exe processes should be running, so no worries there. While I've not observed an instance where a Splunk process will consume 100% of the CPU in the short time I've been monitoring, I do find it unusual to have these other processes start/stop, especially when I've not configured them to do so. I've searched through the Splunk documentation and have not found anything helpful, which is why I'm posting this out to the community.

Has anyone else seen this behavior with servers running the Universal Forwarder? Is there a means by which I can disable these processes from starting? Feel free to point me to any documentation or examples.

Thank you in advance for your time!
-Todd

[helge · Accepted solution]

You can disable by adding interval = -1 to the relevant stanzas. See this answer for details.

[TAE2112]

Ah, a very good point - I should have also posted what I had in place. During the installation, I followed the "custom" installation which (as I discovered) allows for a more granular collection of data. The resulting inputs.conf files (found at C:|Program Files|SplunkUniversalForwarder|etc|apps|Splunk_TA_windows|local) is pasted below for review. I was a little confused as to why the PerfMon stanzas were in place, but I'm guessing it's a common thing to have enabled and is placed there out of convenience. When I saw that the "disabled" value was set to "1" (true), I became less concerned.

[WinEventLog://Application]
disabled = 0
index = xrs_prod_wineventlog

[WinEventLog://Security]

[WinEventLog://System]
disabled = 0
index = xrs_prod_wineventlog

[perfmon://CPUTime]
counters = % Processor Time;% User Time
disabled = 1
index = perfmon
instances = _Total
interval = 10
object = Processor
useEnglishOnly = true

[perfmon://FreeDiskSpace]
counters = Free Megabytes;% Free Space
disabled = 1
index = perfmon
instances = *
interval = 10
object = LogicalDisk
useEnglishOnly = true

[perfmon://LocalNetwork]
counters = Bytes Received/sec;Bytes Sent/sec;Bytes Total/sec;Current Bandwidth
disabled = 1
index = perfmon
instances = *
interval = 10
object = Network Interface
useEnglishOnly = true

Edit: Not sure why the backslashes in my path above were removed, but since they seem to be dropped, I've used the pipe | character to represent the backslash.

[aalanisr26]

Probably when you installed the UF on that server all options were checked durting the installation, so that UF is trying to collect information from Active Directory, Peformance monitor, Network etc.

If you just want to enable windows event logs : System, Security and Application, you need to make sure the rest of the inputs are disabled

go to apps/Splunk_TA_Windows/local/inputs.conf and make sure the additional ones are disabled=true