All Apps and Add-ons

Performance Monitoring not working on windows app?

msarro
Builder

Hey everyone. I have deployed the most recent windows app to my search heads, indexers, and the TA to all of our windows server (all windows 2008 server). In the TA I have added a local/inputs.conf file which contains the following:

...
###### Splunk 5.0+ Performance Counters ######

## CPU
[perfmon://CPU]
counters = *
disabled = 0
instances = *
interval = 10
object = Processor

## Logical Disk
[perfmon://LogicalDisk]
counters = *
disabled = 0
instances = *
interval = 10
object = LogicalDisk

## Physical Disk
[perfmon://PhysicalDisk]
counters = *
disabled = 0
instances = *
interval = 10
object = PhysicalDisk

## Memory
[perfmon://Memory]
counters = *
disabled = 0
interval = 10
object = Memory

## Network
[perfmon://Network]
counters = *
disabled = 0
instances = *
interval = 10
object = Network Interface

## Process
[perfmon://Process]
counters = *
disabled = 0
instances = *
interval = 10
object = Process


## System
[perfmon://System]
counters = *
disabled = 0
instances = *
interval = 10
object = System
...

This is more or less the stock file, just with all of the inputs enabled. I have this deployed to all of our windows servers, but none appear to be sending performance data. Log data is coming in ok, but nothing for performance. All servers are using the v5.0.3 forwarder. I have also tried modifying the inputs.conf file to use the older formatted v4 stanza names, and a separate perfmon.conf file, and I simply cannot get this to work.

I have to use the forwarder because our splunk indexers and search heads are all running linux. Any advice would be very much appreciated on getting this running.

0 Karma
1 Solution

lukejadamec
Super Champion

I managed to recreate the loss of perfmon data with your config, and then fix it.
Your input stanzas are wrong. Try these:

###### Splunk 5.0+ Performance Counters ######

## CPU
[perfmon://Windows__Processor]
counters = *
disabled = 0
instances = *
interval = 10
object = Processor

## Logical Disk
[perfmon://Windows__LogicalDisk]
counters = *
disabled = 0
instances = *
interval = 10
object = LogicalDisk

## Physical Disk
[perfmon://Windows__PhysicalDisk]
counters = *
disabled = 0
instances = *
interval = 10
object = PhysicalDisk

## Memory
[perfmon://Windows__Memory]
counters = *
disabled = 0
interval = 10
object = Memory

## Network
[perfmon://Windows__Network Interface]
counters = *
disabled = 0
instances = *
interval = 10
object = Network Interface

## Process
[perfmon://Windows__Process]
counters = *
disabled = 0
instances = *
interval = 10
object = Process

## System
[perfmon://Windows__System]
counters = *
disabled = 0
instances = *
interval = 10
object = System

Thanks for asking the question. This version of the Windows App is much better than the one I was using. Too bad it took so long to get working.

View solution in original post

lukejadamec
Super Champion

I managed to recreate the loss of perfmon data with your config, and then fix it.
Your input stanzas are wrong. Try these:

###### Splunk 5.0+ Performance Counters ######

## CPU
[perfmon://Windows__Processor]
counters = *
disabled = 0
instances = *
interval = 10
object = Processor

## Logical Disk
[perfmon://Windows__LogicalDisk]
counters = *
disabled = 0
instances = *
interval = 10
object = LogicalDisk

## Physical Disk
[perfmon://Windows__PhysicalDisk]
counters = *
disabled = 0
instances = *
interval = 10
object = PhysicalDisk

## Memory
[perfmon://Windows__Memory]
counters = *
disabled = 0
interval = 10
object = Memory

## Network
[perfmon://Windows__Network Interface]
counters = *
disabled = 0
instances = *
interval = 10
object = Network Interface

## Process
[perfmon://Windows__Process]
counters = *
disabled = 0
instances = *
interval = 10
object = Process

## System
[perfmon://Windows__System]
counters = *
disabled = 0
instances = *
interval = 10
object = System

Thanks for asking the question. This version of the Windows App is much better than the one I was using. Too bad it took so long to get working.

View solution in original post

lukejadamec
Super Champion

The names are preceded by "Windows__" in the working config. With a few other minor changes.

0 Karma

malmoore
Splunk Employee
Splunk Employee

What exactly was wrong? I don't see a difference other than the stanza name.

0 Karma

msarro
Builder

BINGO! That worked beautifully! I will update the support ticket to let them know that the stanzas included in the TA are apparently incorrect. I appreciate your help, thanks!

0 Karma

lukejadamec
Super Champion

On the Performance Management summary page does it say

"Note that these searches depend on indexing WMI data. If that's not enabled in the WMI Inputs section of Manager, you won't see anything below"?

I'm not being a wiseguy, just not sure if they started using perfmon data in the latest version of the App.

Regardless, the message is not entirely accurate. I have it working by enabling WMI for performnance data on the forwarders only.

You should probably not have both perfmon and WMI performance enabled because they both use a lot of index volume.

0 Karma

lukejadamec
Super Champion

The perfmon index will only make it work less. I tried it and the app can't see it. The perfmon data needs to be in the main index.
I also found that it will work with WMI:perfmon* sourcetypes, but I've never seen those. The WMI performance data that I'm familiar with is WMI:CPU or WMI:memory etc...

0 Karma

msarro
Builder

I've added a perfmon index on all of our indexers, and I'm still not seeing any data coming in, so I'm opening a ticket with support to see if they have any feedback in the meantime. Any other ideas in the meantime are appreciated!

0 Karma

lukejadamec
Super Champion

I've installed the latest windows app, and it did not work for me either. but I've found out a few things.
WMI data does not work anymore, the new app uses perfmon data.
You will need to configure the Settings > Lookup Management to get the performance data to populate. See this post: http://answers.splunk.com/answers/109108/questions-regarding-setup-of-splunk-for-windows-app-and-per...
When you configure these settings, you will get no data if the perfmon data does not exist (good test to see if the data is really not there).
Check the logs on the forwarders for perfmon issue

0 Karma

msarro
Builder

Hm, I will try creating the perfmon index on our indexers then. Our search heads are configured to forward everything to indexers and do no local indexing, so every index has to be defined manually. I'll give it a shot.

0 Karma

lukejadamec
Super Champion

As for the permon data, in my version the data is in the perfmon index not the main index.
Try searching index=perfmon

Since the WMI data is being collected locally it does not need anything special to get the data to the indexer, it is simply sent with all of the other log data as data.

0 Karma

msarro
Builder

Negative, it doesn't say that anywhere on the app's Performance Monitoring page. I'm kind of lost since I'm much more of a *nix admin than a windows admin. All of the windows boxes have the forwarder installed, they have the perfmon stanzas, so as far as I can tell, that should be everything needed right? The documentation for the app only discusses remote monitoring which I can't do with *nix splunk indexers and search heads, plus this is across numerous domains so I don't know how well wmi would work.

0 Karma

msarro
Builder

In the windows app, when you click on the "Performance monitoring" tab, your receive a notice that "no matching fields exist", and none of the searches return results. If you use search and look in the main index, the only sources that are coming in from our windows boxes are WinEventLog:Application, WinEventLog:Security, and WinEventLog:System. Nothing related to performance.

0 Karma

lukejadamec
Super Champion

When you say there is no performance data are you referring to the Windows App or a search that looks for perfmon data?

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.