Getting Data In

Why are these additional Splunk processes starting and stopping on Windows hosts configured with Universal Forwarders?

TAE2112
Explorer

I have a collection of Windows 2008R2 servers running the Universal Forwarder which I configured to forward Windows Event Logs (Application and System) to a specific index on a dedicated Splunk Indexer server. Because I'm pretty new to Splunk, I'm of the opinion that it's a pretty basic setup, with nothing really fancy going on. I simply installed the Universal Forwarder and (eventually) figured out how to collect the Event logs and send them to the Indexer.

Over the past few weeks, I've been chasing an issue with CPU utilization on a few of these servers (the ones in the "collection") with little luck. The Development group has observed that the process "splunk-admon.exe" has been consuming up to 100% of the CPU on servers where the Universal Forwarders are installed. After sitting on one of the problem servers for a while, I've noticed that in addition to the persistent processes, there are a few Splunk-related processes which keep popping up and then closing:

splunkd.exe (constantly running)
splunk-winevtlog.exe (constantly running)
splunk-MonitorNoHandle.exe (start/stop)
splunk-winprintmon.exe (start/stop)
splunk-netmon.exe (start/stop)
splunk-admon.exe (start/stop)
splunk-regmon.exe (start/stop)

I expected that the splunkd.exe and splunk-winevtlog.exe processes should be running, so no worries there. While I've not observed an instance where a Splunk process will consume 100% of the CPU in the short time I've been monitoring, I do find it unusual to have these other processes start/stop, especially when I've not configured them to do so. I've searched through the Splunk documentation and have not found anything helpful, which is why I'm posting this out to the community.

Has anyone else seen this behavior with servers running the Universal Forwarder? Is there a means by which I can disable these processes from starting? Feel free to point me to any documentation or examples.

Thank you in advance for your time!
-Todd

1 Solution

helge
Builder

You can disable by adding interval = -1 to the relevant stanzas. See this answer for details.

View solution in original post

helge
Builder

You can disable by adding interval = -1 to the relevant stanzas. See this answer for details.

TAE2112
Explorer

Ah, a very good point - I should have also posted what I had in place. During the installation, I followed the "custom" installation which (as I discovered) allows for a more granular collection of data. The resulting inputs.conf files (found at C:|Program Files|SplunkUniversalForwarder|etc|apps|Splunk_TA_windows|local) is pasted below for review. I was a little confused as to why the PerfMon stanzas were in place, but I'm guessing it's a common thing to have enabled and is placed there out of convenience. When I saw that the "disabled" value was set to "1" (true), I became less concerned.

[WinEventLog://Application]
disabled = 0
index = xrs_prod_wineventlog

[WinEventLog://Security]

[WinEventLog://System]
disabled = 0
index = xrs_prod_wineventlog

[perfmon://CPUTime]
counters = % Processor Time;% User Time
disabled = 1
index = perfmon
instances = _Total
interval = 10
object = Processor
useEnglishOnly = true

[perfmon://FreeDiskSpace]
counters = Free Megabytes;% Free Space
disabled = 1
index = perfmon
instances = *
interval = 10
object = LogicalDisk
useEnglishOnly = true

[perfmon://LocalNetwork]
counters = Bytes Received/sec;Bytes Sent/sec;Bytes Total/sec;Current Bandwidth
disabled = 1
index = perfmon
instances = *
interval = 10
object = Network Interface
useEnglishOnly = true

Edit: Not sure why the backslashes in my path above were removed, but since they seem to be dropped, I've used the pipe | character to represent the backslash.

0 Karma

aalanisr26
Path Finder

Probably when you installed the UF on that server all options were checked durting the installation, so that UF is trying to collect information from Active Directory, Peformance monitor, Network etc.

If you just want to enable windows event logs : System, Security and Application, you need to make sure the rest of the inputs are disabled

go to apps/Splunk_TA_Windows/local/inputs.conf and make sure the additional ones are disabled=true

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...