What's the best way to get Windows Perfmon data into a Metrics Index?

[Tohrment]

The question pretty much sums it up.

I am wanting to get PerfMon data into a Metrics index and have been banging my head against it for about a week now. So far, I have been unsuccessful in my endeavor.

I was attempting to get it directly from the universal forwarder, but apparently the data is not formatted properly. I am pretty new to the wide world of Splunk, and have been reading anything I can find even slightly related to this, but there doesn't seem to be much out there as of now.

If there is a better way to do it? I am all ears(or eyes in this case).

Thanks Splunkers.

[Tohrment · Accepted solution]

Ok, so here is where we went wrong and why we weren't getting any metrics in.

1.) INSTALL THE ADD-ON to ALL layers to ensure the proper transforms occur. I was unaware it wasn't on our HF or indexers(we have a guy that runs all that side of things).

2.) Following gjanders link confirmed that I had set up the proper local/inputs.conf stanzas but due to #1 not being done we were not getting data.
Collect perfmon data and wmi:uptime data in metric index

3.) Make sure the entire stack is up to date(at the time of this writing it is 7.2)

Thank you gjanders for pointing me to that link. Literally showed me the last bolt I was missing to make this thing work!

[sloshburch]

If it helps: What is the best way to migrate Windows performance monitoring from event-based to metrics-based dat...

[gjanders]

Can you confirm your following Configure the Splunk Add-on for Windows, in particular Collect perfmon data and wmi:uptime data in metric index

And as per Compatibility between forwarders and Splunk Enterprise indexers you are using the 7.x or above universal forwarder and indexer?

[Tohrment]

I have it setup precisely how that is set up and it is giving me the following error

Search peer 1 has the following message: Index Processor: Metric name is missing for source=Process, sourcetype=Process, host=servername, index=winmetrics. Metric event data without metric name is invalid and would not be indexed. Ensure the input metric data is not malformed.

I have it set up exactly how the first link you gave me is set up. Do you have this working?

[gjanders]

Converted to tn answer so you can either accept this one or accept your answer.

Glad I could help!

[sudosplunk]

Hi there, are you using the app built by splunk1 to collect perfmon data from windows? Also, have a look at splunk docs on perfmon2 to get some more insights. HTH!

[Tohrment]

We use the Splunk add-on for Windows and it has the perfmon stanzas in the inputs.conf. Those however do not provide the data in a way that the Metrics index will accept. What I am aiming to do is create the stack in the Metrics Workspace to be able to dig down through related objects when an issue arises. I have been unable to find anything other than "Getting Data in from other Sources" and it talks about HEC and metrics-csv inputs but no links to best practices on creating the CSVs for ingest to the Metrics index.

Basically, what the question is based around is what is the best way to create the csv or transform the perfmon data into a csv for ingest into the metrics index. If that is not the best way to go about it than what is the suggested path to get that data in a format that a metrics index will accept?

Hopefully that clears up what this question is really about.

[sudosplunk]

Ah, got it. Thanks for the clarification. Have a look at "maciep's" answer in the below link and see if it something that will meet your requirement. If you've already tried that approach, then pls let me know what are the issues you're facing.

https://answers.splunk.com/answers/607304/sending-perfmon-data-to-metrics-index.html

[Tohrment]

so, that did not work. Testing some other transforms in the hopes of stumbling my way into an answer lol

[ntankersley_spl]

Add-on for Windows Infrastructure 5.0.0 and later supports metrics transformations. You can also use Windows TA 4.8.4 with Splunk App & Add-on for Infrastructure to accomplish the same thing.

[Tohrment]

I have looked at that but without anyone confirming whether it worked or not made me skeptical. I will talk with my props\transforms guy and see if he thinks it would work. Thanks for all the help and I will report back if it works.