All Apps and Add-ons

Why does Windows perfmon event data stop working after adding the Splunk Add-on for Infrastructure?

sloshburch
Splunk Employee
Splunk Employee

I've been collecting data using Splunk's perfmon modular input as per Splunk Add-on for Microsoft Windows. Yet after I deployed the Splunk Add-on for Infrastructure, I no longer see any event data.

I am confident it is related to the Splunk Add-on for Infrastructure because when I remove that add-on from my indexers, the data gets indexed again.

What gives? Why is the event data missing when the Splunk Add-on for Infrastructure is deployed to my indexers?

1 Solution

sloshburch
Splunk Employee
Splunk Employee

Short Answer

The Splunk Add-on for Infrastructure has configuration that is trying to convert the log event data into metrics data.

Explanation

The Splunk Add-on for Infrastructure is configured to convert any log event data with a source value of Perfmon:* to metrics data. Out of the box, Splunk's perfmon modular input will use Perfmon:* as the source value. So the Splunk Add-on for Infrastructure is trying to convert this data, even if you intended it to stay as log event data.

If you have Splunk Add-on for Microsoft Windows deployed, this should not be an issue because the default source value of the add-on's perfmon data is PerfmonMk:*. Many years ago, Splunk introduced the mode attribute with the value multikv. This value prints events formatted with multiple key-value pairs, not as single line events. This became the default in Splunk Add-on for Microsoft Windows.

Work Around 1: Upgrade Splunk Add-on for Microsoft Windows

If you upgrade Splunk Add-on for Microsoft Windows, the mode = multikv default will take affect and provide the same field extractions your used to.

Any custom knowledge objects that have the source hard coded as Perfmon:* would need to be adjusted to use PerfmonMk:*.

Work Around 2: Custom metrics source value

Follow the pattern of the Splunk Add-on for Infrastructure. Set a custom source value within the perfmon stanzas of inputs.conf and set the log-to-metric conversion to match that customer source value.

The source value can be hardcoded within the inputs.conf sent to each Windows endpoint that runs the perfmon input. For example:

[perfmon://CPU Load]
counters = % C1 Time;% C2 Time;% Idle Time;% Processor Time;% User Time;% Privileged Time;% Reserved Time;% Interrupt Time
instances = *
interval = 60
object = Processor
index = em_metrics
_meta =  env::lab entity_type::Windows_Hos
source = PerfmonMetric:CPU

Then deploy some related props.conf and transforms.conf in accordance to Where to install the App, Add-ons, and other dependencies for the Splunk® App for Infrastructure. You'll be creating a new custom stanza based on the source value used within the inputs.conf. That stanza's attributes can be a copy of the Splunk Add-on for Infrastructure equivalent. In our example:

[PerfmonMetric:CPU]
TRANSFORMS-_value_for_perfmon_metrics_store = value_for_perfmon_metrics_store
TRANSFORMS-metric_name_for_perfmon_metrics_store = metric_name_for_perfmon_metrics_store
TRANSFORMS-object_for_perfmon_metrics_store = object_for_perfmon_metrics_store
TRANSFORMS-instance_for_perfmon_metrics_store = instance_for_perfmon_metrics_store
TRANSFORMS-collection_for_perfmon_metrics_store = collection_for_perfmon_metrics_store
EVAL-metric_type = "gauge"
SEDCMD-remove-whitespace = s/ /_/g s/\s/ /g

Each TRANSFORMS attribute refers to a stanza within the Splunk Add-on for Infrastructure's transforms.conf. So make sure that file is co-located.

View solution in original post

sloshburch
Splunk Employee
Splunk Employee

Short Answer

The Splunk Add-on for Infrastructure has configuration that is trying to convert the log event data into metrics data.

Explanation

The Splunk Add-on for Infrastructure is configured to convert any log event data with a source value of Perfmon:* to metrics data. Out of the box, Splunk's perfmon modular input will use Perfmon:* as the source value. So the Splunk Add-on for Infrastructure is trying to convert this data, even if you intended it to stay as log event data.

If you have Splunk Add-on for Microsoft Windows deployed, this should not be an issue because the default source value of the add-on's perfmon data is PerfmonMk:*. Many years ago, Splunk introduced the mode attribute with the value multikv. This value prints events formatted with multiple key-value pairs, not as single line events. This became the default in Splunk Add-on for Microsoft Windows.

Work Around 1: Upgrade Splunk Add-on for Microsoft Windows

If you upgrade Splunk Add-on for Microsoft Windows, the mode = multikv default will take affect and provide the same field extractions your used to.

Any custom knowledge objects that have the source hard coded as Perfmon:* would need to be adjusted to use PerfmonMk:*.

Work Around 2: Custom metrics source value

Follow the pattern of the Splunk Add-on for Infrastructure. Set a custom source value within the perfmon stanzas of inputs.conf and set the log-to-metric conversion to match that customer source value.

The source value can be hardcoded within the inputs.conf sent to each Windows endpoint that runs the perfmon input. For example:

[perfmon://CPU Load]
counters = % C1 Time;% C2 Time;% Idle Time;% Processor Time;% User Time;% Privileged Time;% Reserved Time;% Interrupt Time
instances = *
interval = 60
object = Processor
index = em_metrics
_meta =  env::lab entity_type::Windows_Hos
source = PerfmonMetric:CPU

Then deploy some related props.conf and transforms.conf in accordance to Where to install the App, Add-ons, and other dependencies for the Splunk® App for Infrastructure. You'll be creating a new custom stanza based on the source value used within the inputs.conf. That stanza's attributes can be a copy of the Splunk Add-on for Infrastructure equivalent. In our example:

[PerfmonMetric:CPU]
TRANSFORMS-_value_for_perfmon_metrics_store = value_for_perfmon_metrics_store
TRANSFORMS-metric_name_for_perfmon_metrics_store = metric_name_for_perfmon_metrics_store
TRANSFORMS-object_for_perfmon_metrics_store = object_for_perfmon_metrics_store
TRANSFORMS-instance_for_perfmon_metrics_store = instance_for_perfmon_metrics_store
TRANSFORMS-collection_for_perfmon_metrics_store = collection_for_perfmon_metrics_store
EVAL-metric_type = "gauge"
SEDCMD-remove-whitespace = s/ /_/g s/\s/ /g

Each TRANSFORMS attribute refers to a stanza within the Splunk Add-on for Infrastructure's transforms.conf. So make sure that file is co-located.

Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...