Getting Data In

How to disable PerfMON stats from remote windows Universal Forwarder?

neerajshah81
Path Finder

Hi All,

I have a single instance Splunk 7.1.2 on Windows platform. I am getting lot of events related to Perfmon:Network_Interface , Perfmon:Memory from every Windows hosts that i have UF installed on. despite not having anything Enabled under ( Web UI >> Settings >> Data Inputs >> Remote performance Monitoring ) . Every counter or input under here as in (Settings >> Data Inputs >> Remote performance Monitoring ) is disabled.

I am using Splunk Windows App for Infrastructure & as part of this, i had to install the TA for Windows on all the Windows clients.
The local inputs.conf ( \etc\apps\Splunk_TA_windows\local ) on each Win client does not have any input stanzas related to perfmon: defined. Below is copy of my inputs.conf

[WinEventLog://Security]
checkpointInterval = 5
current_only = 0
 disabled = 0
 start_from = oldest
 suppress_text = 1

[WinEventLog://System]
checkpointInterval = 5
current_only = 0
disabled = 0
start_from = oldest
 suppress_text = 1

I have checked this post, there is no perfmon.conf file under my C:\SplunkUniversalForwarder\etc\system\default directory on each Windows client. What is then making Splunk ES collect or fetch these perfmon related stats from and how to disable it ?

alt text

alt text

Tags (3)
0 Karma
1 Solution

neerajshah81
Path Finder

Hi All, FYI i ended up opening a support ticket. The cause of the perform stats was etc/apps/Splunk_TA_microsoft_ad/default/inputs.conf & not etc/apps/Splunk_TA_windows/default/inputs.conf which i was suspecting all the while. If you are using the "App for Windows Infrastructure", this in turn needs more add-ons such as "TA for Windows" & "TA for Active Directory" .

The /Splunk_TA_microsoft_ad/default/inputs.conf has some inputs enabled by default and we need to create a local inputs.conf to turn them off. If you are not using the "TA for AD" but just the TA for Windows then you would focus on the "TA for Windows" inputs.conf

View solution in original post

papaleo
New Member

I have the same issues. In my deployment server i put a local inputs.conf and disable perfmon, but I still receive data.
I had to put the same settings in Splunk_TA_windows.
Now I still receive perfmon CPU Load and perfrom Available Memory, but in main index.
I cannot figure out where to disable, because I disable both in Splunk_TA_windows and Splunk_TA_windows_AD, but I still receive those perfmon metrics in main index (even if I defined perfmon index for those metrics).
I think I will open a ticket.

0 Karma

CarsonZa
Contributor

@papaleo,

you should really create a new post if you want an answer but ill go ahead and give you a nudge in the right direction.

run ./splunk cmd btool inputs list --debug on one of your forwarders that still sending perfmon data. That will tell you all the input stanzas and where the config file for that stanza is.

0 Karma

papaleo
New Member

Thank you very much for the quick reply, is very useful your hint!
I solved my issue, adding the following line in both inputs.conf in Windows and Microsoft_AD add-on.

[perfmon://CPU Load]
disabled = 1
index = perfmon
instances = *
object = Processor
useEnglishOnly = true

[perfmon://Memory Available]
disabled = 1
object = Memory
instances = *
useEnglishOnly = true
index = perfmon

0 Karma

neerajshah81
Path Finder

Hi All, FYI i ended up opening a support ticket. The cause of the perform stats was etc/apps/Splunk_TA_microsoft_ad/default/inputs.conf & not etc/apps/Splunk_TA_windows/default/inputs.conf which i was suspecting all the while. If you are using the "App for Windows Infrastructure", this in turn needs more add-ons such as "TA for Windows" & "TA for Active Directory" .

The /Splunk_TA_microsoft_ad/default/inputs.conf has some inputs enabled by default and we need to create a local inputs.conf to turn them off. If you are not using the "TA for AD" but just the TA for Windows then you would focus on the "TA for Windows" inputs.conf

sudosplunk
Motivator

Hello @neerajshah81,

Splunk on Windows ships with several Windows-only inputs. They are defined in the default inputs.conf and are enabled by default.
You should explicitly define the monitor input like below, under etc\apps\Splunk_TA_windows\default\inputs.conf, to overwrite default settings. I've had the same issue in the past.

You can find more information on default settings here.

[perfmon://<name>]
disabled=1
0 Karma

neerajshah81
Path Finder

@nittala_surya, isn't that redundant work ? i have re-checked , every input is set to disabled =1 in \etc\apps\Splunk_TA_windows\default\inputs.conf by default. Are you guys suggesting despite of this, we need to add those inputs in local\inputs.conf and again set them to disabled =1 ?

What is causing Splunk on Windows to enable them in the first place then ?

0 Karma

sudosplunk
Motivator

No. In \etc\apps\Splunk_TA_windows\default\inputs.conf, perfmon input is not explicitly disabled which means, no one is telling splunk to disable 'perform' at higher precedence level and hence, system/default(least precedence) is applied to these inputs.
In short, yes, you should add the input and disable it at a higher directory structure.

neerajshah81
Path Finder

@ nittala_surya, it turns out the cause of problem was etc/apps/Splunk_TA_microsoft_ad/default/inputs.conf & not etc/apps/Splunk_TA_windows/default/inputs.conf . So i had to create a local inputs.conf inside the TA_microsoft_AD folder. This TA for AD also comes into picture when you install the "Splunk App for Windows Infrastructure". I again confirmed with support, that perfmon stats are disabled by default in "Splunk_TA_Windows" which was the source of confusion earlier.

0 Karma

neerajshah81
Path Finder

Thank you very much. Let me give that a shot and get back.

0 Karma

CarsonZa
Contributor

which is exactly what ive said above

neerajshah81
Path Finder

@CarsonZa , thanks for your help. I have granted you UP votes. I have posted the answer what worked in "my" case.

0 Karma

CarsonZa
Contributor

check the inputs.conf under default. Ive also seen some perfmon inputs under system/local. Check those and create a stanza in your local directory disabling each stanza

neerajshah81
Path Finder

HI Carson, Which "default" folder are you referring to for checking inputs.conf ? There are cpl of them : etc\apps\Splunk_TA_windows\default and etc\system\default

Regarding, /etc/system/local/inputs.conf it does not have anything other than [Host] .

0 Karma

rmjharris
Path Finder

Never edit anything in the default folder.

Copy etc\apps\Splunk_TA_windows\default\inputs.conf to etc\apps\Splunk_TA_windows\local. Then edit disabled = 0 or 1 for each stanza.

Doc is here.

http://docs.splunk.com/Documentation/WindowsAddOn/5.0.0/User/Configuration

neerajshah81
Path Finder

Hello rmjharris, i agree i did not edit anything there. However, everything (every input) under etc\apps\Splunk_TA_windows\default\inputs.conf is set to disabled =1 by default. So there should be nothing more to do as in there shouldn't be any need for us to re define the perfmon inputs again in the local\inputs.conf when they are already disabled by default, correct ?
Not sure what is causing Splunk to collect those perfmon stats then.

0 Karma

CarsonZa
Contributor

@rmjharris i never said edit the conf in the default folder is said, check it. i have had several instances where the default has things enabled and the only way to check them to see if theyre disable is to look in the default folder to see whats enabled and whats not. Then follow my instructions which are identical to yours.

0 Karma

rmjharris
Path Finder

You are correct, I misread.

0 Karma

rmjharris
Path Finder

You're right, it should be disabled. Is the screenshot correct that these events are only coming from two hosts? How many Windows hosts have the TA installed?

0 Karma

neerajshah81
Path Finder

Correct, there are only 2 hosts that we are monitoring at the moment. This is because we just setup Splunk 3 - 4 days back. Both are Windows AD domain controllers. The plan is to get Windows clients configured correctly & then move on to Linux clients which are a handful.

Any further troubleshooting advise will be appreciated. I am going to open up a support ticket as well. This is driving me nuts. Those perfom stats chew up the license usage.

0 Karma

CarsonZa
Contributor

\etc\apps\Splunk_TA_windows\default\inputs.conf

every stanza in here should be set to disabled = 1 or true

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...