Splunk sub-processes start/stop every minute (splunk-admon, splunk-powershell, etc). How do we prevent this?

[hortonew]

Is there a way to disable these processes from ever starting? I've tried with a config to overwrite what they get from $SPLUNK_HOME$/etc/system/default/inputs.conf, but I think it might only be possible to specify a really long interval, as by default they have an interval of 60. Thoughts?

[admon]
disabled = 1

[WinNetMon]
disabled = 1

[WinPrintMon]
disabled = 1

[WinRegMon]
disabled = 1

[perfmon]
disabled = 1

[powershell]
disabled = 1

[powershell2]
disabled = 1

[rkantamaneni_sp]

This is bug SPL-145668 in Splunk. Note that this is now fixed in Splunk 7.3.0 (it's not yet in the Release Notes, support is working on getting it published there and in the Known Issues for all prior versions of Splunk).

disabled = 1 should be respected appropriately for Splunk Windows processes moving forward.

[michaeljorgense]

@rkantamaneni_sp  are you talking about this being specifically fixed in Splunk Universal Forwarder version 7.3.0? i.e. would just patching the Windows UFs to use 7.3.0 fix this issue? Or does it need 7.3.0 in the server end too?

Also I notice there isn't any reference to SPL-145668 in the Splunk Universal Forwarder release notes yet.

[rkantamaneni_sp]

Hi @michaeljorgense ,

>> are you talking about this being specifically fixed in Splunk Universal Forwarder version 7.3.0? i.e. would just patching the Windows UFs to use 7.3.0 fix this issue? Or does it need 7.3.0 in the server end too?

This issue would be for either Splunk instance (UF or Enterprise). The issue has to do with the running instance of Splunk (where you'd see the problem).

In case you can't upgrade at the moment, the workaround is:

To mitigate the calls on instances that have inputs, set disable = -1 to run the process once on startup (but the process will still be running).

To correct the issue on indexers or instances on Windows that don't have any inputs, the default inputs.conf and the inputs.conf.spec files can be removed so that introspection does not try to run the script. Contact Splunk Support or your Splunk Account team for assistance.

>> Also I notice there isn't any reference to SPL-145668 in the Splunk Universal Forwarder release notes yet.

It looks like this is not in the release notes for the UF of Splunk Enterprise as of this moment, I've gone ahead and placed an inquiry to review if it should be there.

[nick405060]

The answer posted here works, but not if you have Splunk_TA_windows installed. For us, we had a full CPU taken up by splunk-netmon.exe on our Exchange server, that wouldn't get disabled if you set "interval = -1" or "disabled = 1" in etc/system/local/inputs.conf.

So I ran "./splunk cmd btool inputs list --debug | select-string netmon" and saw that I actually had to modify C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf, and set the two stanzas there to "disabled = 1". That did the trick!

[vaneet]

Which two stanzas?

[rkantamaneni_sp]

You probably need to run the btool command as mentioned:

"./splunk cmd btool inputs list --debug | select-string netmon"

[jtacy · Accepted solution]

I had the same dilemma for an application that required only Windows event log inputs and decided to make the modular input processes run just once by using the interval = -1 setting on the inputs. For example, an inputs.conf that will allow only the WinEventLog modular input to attempt to restart itself on a 6.4.1 install looks like this:

[admon]
interval = -1

[WinNetMon]
interval = -1

[WinPrintMon]
interval = -1

[WinRegMon]
interval = -1

[MonitorNoHandle]
interval = -1

[powershell]
interval = -1

[powershell2]
interval = -1

You'll end up with the following intervals according to splunkd.log:

INFO  ExecProcessor - New scheduled exec process: "C:\Program Files\SplunkUniversalForwarder\bin\splunk-wmi.exe"
INFO  ExecProcessor -   interval: 10000000000 ms
INFO  ExecProcessor - New scheduled exec process: "C:\Program Files\SplunkUniversalForwarder\bin\splunk-MonitorNoHandle.exe"
INFO  ExecProcessor -   interval: run once
INFO  ExecProcessor - New scheduled exec process: "C:\Program Files\SplunkUniversalForwarder\bin\splunk-admon.exe"
INFO  ExecProcessor -   interval: run once
INFO  ExecProcessor - New scheduled exec process: "C:\Program Files\SplunkUniversalForwarder\bin\splunk-netmon.exe"
INFO  ExecProcessor -   interval: run once
INFO  ExecProcessor - New scheduled exec process: "C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe"
INFO  ExecProcessor -   interval: run once
INFO  ExecProcessor - New scheduled exec process: "C:\Program Files\SplunkUniversalForwarder\bin\splunk-powershell.exe"
INFO  ExecProcessor -   interval: run once
INFO  ExecProcessor - New scheduled exec process: "C:\Program Files\SplunkUniversalForwarder\bin\splunk-powershell.exe" --ps2
INFO  ExecProcessor -   interval: run once
INFO  ExecProcessor - New scheduled exec process: "C:\Program Files\SplunkUniversalForwarder\bin\splunk-regmon.exe"
INFO  ExecProcessor -   interval: run once
INFO  ExecProcessor - New scheduled exec process: "C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe"
INFO  ExecProcessor -   interval: 60000 ms
INFO  ExecProcessor - New scheduled exec process: "C:\Program Files\SplunkUniversalForwarder\bin\splunk-winprintmon.exe"
INFO  ExecProcessor -   interval: run once

This configuration has worked well with over 10,000 machines. Note that the splunk-winevtlog.exe interval is still 60s but it won't actually start a new process unless the running process fails for some reason. Also note that splunk-wmi.exe will still run about every 4 months with the above configuration. If you have a problem with that you could probably try the same approach with that scripted input:

[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path]
interval = -1

Have fun!

[hortonew]

Thanks - We had actually worked through this in IRC, but appreciate the response. It does work great. I just put this in an app's default directory so that other app's local can take precedence.

[shocko]

Same issue here guys. Thanks for the post.

[hortonew]

More info: https://answers.splunk.com/answers/206021/windows-universal-forwarder-default-scripted-input.html