Getting Data In

Windows Perfmon Collection Issue

nickkoe
Explorer

Having some issues with collecting % Processor Time for processes. My inputs.conf is configured with the below stanza:

[perfmon://Process]
counters = % Processor Time; etc.
instances = *
disabled = 0
interval = 600
object = Process
sourcetype = Process
index = Test

The server has roughly 63 processes going at anytime and for most counters, I get that many instances returned when I search. However, for % Processor Time I cant seem to get back more than 18 instances. And if I bounce Splunk on the forwarder I get back a different number of instances every time.

Anyone else have this issue when trying to collect % Processor Time for Processes? Thanks!

0 Karma
1 Solution

wyfwa4
Communicator

By default, Splunk drops zero value perfmon data and so you get gaps in your data - this can be modified by using the ShowZeroValue option in the input stanza. This article explains this issue in more details

http://blogs.splunk.com/2013/10/28/new-features-for-perfmon-in-splunk-6/

I have found perfmon collecting from the 6.3.1 forwarder to be unstable and get data drops quite often when other counters are working ok through Splunk. I am assuming this is due to the amount of data being collected compared to other counters, but still seems strange (maybe a bug?) So you need to monitor this behaviour and make sure the data feed is stable.

I have modified my perfmon collection to use the new MK counters detailed in the same article and found they save a significant amount of space compared to the standard data format - this is especially the case for process data, where you may have hundreds of processes running concurrently. The events themselves are not easy to understand, but the data is automatically extracted to the relevant fields and so still easy enough to manipulate.

View solution in original post

wyfwa4
Communicator

By default, Splunk drops zero value perfmon data and so you get gaps in your data - this can be modified by using the ShowZeroValue option in the input stanza. This article explains this issue in more details

http://blogs.splunk.com/2013/10/28/new-features-for-perfmon-in-splunk-6/

I have found perfmon collecting from the 6.3.1 forwarder to be unstable and get data drops quite often when other counters are working ok through Splunk. I am assuming this is due to the amount of data being collected compared to other counters, but still seems strange (maybe a bug?) So you need to monitor this behaviour and make sure the data feed is stable.

I have modified my perfmon collection to use the new MK counters detailed in the same article and found they save a significant amount of space compared to the standard data format - this is especially the case for process data, where you may have hundreds of processes running concurrently. The events themselves are not easy to understand, but the data is automatically extracted to the relevant fields and so still easy enough to manipulate.

nickkoe
Explorer

Works great, thanks!

0 Karma

nickkoe
Explorer

After some testing, and probably listed, somewhere in the documentation. It appears that it will only report back on a process that has CPU usage during the pull. Memory and other things will always return as the system always reserve some memory for a process. Again, it makes since, for peace of mind I was still hoping it would return a value for every process. I could be wrong but this seems to be the explanation.

0 Karma

rjthibod
Champion

Can you clarify what version of the Splunk Forwarder you are running and the type of Windows system on which it is running?

0 Karma

nickkoe
Explorer

O sorry, Its 6.4.3 and this is on a 2008 server, same issue on 2012 though.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...