How to disable PerfMON stats from remote windows Universal Forwarder?

[neerajshah81]

Hi All,

I have a single instance Splunk 7.1.2 on Windows platform. I am getting lot of events related to Perfmon:Network_Interface , Perfmon:Memory from every Windows hosts that i have UF installed on. despite not having anything Enabled under ( Web UI >> Settings >> Data Inputs >> Remote performance Monitoring ) . Every counter or input under here as in (Settings >> Data Inputs >> Remote performance Monitoring ) is disabled.

I am using Splunk Windows App for Infrastructure & as part of this, i had to install the TA for Windows on all the Windows clients.
The local inputs.conf ( \etc\apps\Splunk_TA_windows\local ) on each Win client does not have any input stanzas related to perfmon: defined. Below is copy of my inputs.conf

[WinEventLog://Security]
checkpointInterval = 5
current_only = 0
 disabled = 0
 start_from = oldest
 suppress_text = 1

[WinEventLog://System]
checkpointInterval = 5
current_only = 0
disabled = 0
start_from = oldest
 suppress_text = 1

I have checked this post, there is no perfmon.conf file under my C:\SplunkUniversalForwarder\etc\system\default directory on each Windows client. What is then making Splunk ES collect or fetch these perfmon related stats from and how to disable it ?

[papaleo]

I have the same issues. In my deployment server i put a local inputs.conf and disable perfmon, but I still receive data.
I had to put the same settings in Splunk_TA_windows.
Now I still receive perfmon CPU Load and perfrom Available Memory, but in main index.
I cannot figure out where to disable, because I disable both in Splunk_TA_windows and Splunk_TA_windows_AD, but I still receive those perfmon metrics in main index (even if I defined perfmon index for those metrics).
I think I will open a ticket.

[CarsonZa]

@papaleo,

you should really create a new post if you want an answer but ill go ahead and give you a nudge in the right direction.

run ./splunk cmd btool inputs list --debug on one of your forwarders that still sending perfmon data. That will tell you all the input stanzas and where the config file for that stanza is.

[papaleo]

Thank you very much for the quick reply, is very useful your hint!
I solved my issue, adding the following line in both inputs.conf in Windows and Microsoft_AD add-on.

[perfmon://CPU Load]
disabled = 1
index = perfmon
instances = *
object = Processor
useEnglishOnly = true

[perfmon://Memory Available]
disabled = 1
object = Memory
instances = *
useEnglishOnly = true
index = perfmon

[neerajshah81 · Accepted solution]

Hi All, FYI i ended up opening a support ticket. The cause of the perform stats was etc/apps/Splunk_TA_microsoft_ad/default/inputs.conf & not etc/apps/Splunk_TA_windows/default/inputs.conf which i was suspecting all the while. If you are using the "App for Windows Infrastructure", this in turn needs more add-ons such as "TA for Windows" & "TA for Active Directory" .

The /Splunk_TA_microsoft_ad/default/inputs.conf has some inputs enabled by default and we need to create a local inputs.conf to turn them off. If you are not using the "TA for AD" but just the TA for Windows then you would focus on the "TA for Windows" inputs.conf

[sudosplunk]

Hello @neerajshah81,

Splunk on Windows ships with several Windows-only inputs. They are defined in the default inputs.conf and are enabled by default.
You should explicitly define the monitor input like below, under etc\apps\Splunk_TA_windows\default\inputs.conf, to overwrite default settings. I've had the same issue in the past.

You can find more information on default settings here.

[perfmon://<name>]
disabled=1

[neerajshah81]

@Anonymous, isn't that redundant work ? i have re-checked , every input is set to disabled =1 in \etc\apps\Splunk_TA_windows\default\inputs.conf by default. Are you guys suggesting despite of this, we need to add those inputs in local\inputs.conf and again set them to disabled =1 ?

What is causing Splunk on Windows to enable them in the first place then ?

[sudosplunk]

No. In \etc\apps\Splunk_TA_windows\default\inputs.conf, perfmon input is not explicitly disabled which means, no one is telling splunk to disable 'perform' at higher precedence level and hence, system/default(least precedence) is applied to these inputs.
In short, yes, you should add the input and disable it at a higher directory structure.

[neerajshah81]

@ nittala_surya, it turns out the cause of problem was etc/apps/Splunk_TA_microsoft_ad/default/inputs.conf & not etc/apps/Splunk_TA_windows/default/inputs.conf . So i had to create a local inputs.conf inside the TA_microsoft_AD folder. This TA for AD also comes into picture when you install the "Splunk App for Windows Infrastructure". I again confirmed with support, that perfmon stats are disabled by default in "Splunk_TA_Windows" which was the source of confusion earlier.

[neerajshah81]

Thank you very much. Let me give that a shot and get back.

[CarsonZa]

which is exactly what ive said above

[neerajshah81]

@CarsonZa , thanks for your help. I have granted you UP votes. I have posted the answer what worked in "my" case.

[CarsonZa]

check the inputs.conf under default. Ive also seen some perfmon inputs under system/local. Check those and create a stanza in your local directory disabling each stanza

[neerajshah81]

HI Carson, Which "default" folder are you referring to for checking inputs.conf ? There are cpl of them : etc\apps\Splunk_TA_windows\default and etc\system\default

Regarding, /etc/system/local/inputs.conf it does not have anything other than [Host] .

[rmjharris]

Never edit anything in the default folder.

Copy etc\apps\Splunk_TA_windows\default\inputs.conf to etc\apps\Splunk_TA_windows\local. Then edit disabled = 0 or 1 for each stanza.

Doc is here.

http://docs.splunk.com/Documentation/WindowsAddOn/5.0.0/User/Configuration

[neerajshah81]

Hello rmjharris, i agree i did not edit anything there. However, everything (every input) under etc\apps\Splunk_TA_windows\default\inputs.conf is set to disabled =1 by default. So there should be nothing more to do as in there shouldn't be any need for us to re define the perfmon inputs again in the local\inputs.conf when they are already disabled by default, correct ?
Not sure what is causing Splunk to collect those perfmon stats then.

[CarsonZa]

@rmjharris i never said edit the conf in the default folder is said, check it. i have had several instances where the default has things enabled and the only way to check them to see if theyre disable is to look in the default folder to see whats enabled and whats not. Then follow my instructions which are identical to yours.

[rmjharris]

You are correct, I misread.

[rmjharris]

You're right, it should be disabled. Is the screenshot correct that these events are only coming from two hosts? How many Windows hosts have the TA installed?

[neerajshah81]

Correct, there are only 2 hosts that we are monitoring at the moment. This is because we just setup Splunk 3 - 4 days back. Both are Windows AD domain controllers. The plan is to get Windows clients configured correctly & then move on to Linux clients which are a handful.

Any further troubleshooting advise will be appreciated. I am going to open up a support ticket as well. This is driving me nuts. Those perfom stats chew up the license usage.

[CarsonZa]

\etc\apps\Splunk_TA_windows\default\inputs.conf

every stanza in here should be set to disabled = 1 or true