Why does Windows perfmon event data stop working after adding the Splunk Add-on for Infrastructure?

[sloshburch]

I've been collecting data using Splunk's perfmon modular input as per Splunk Add-on for Microsoft Windows. Yet after I deployed the Splunk Add-on for Infrastructure, I no longer see any event data.

I am confident it is related to the Splunk Add-on for Infrastructure because when I remove that add-on from my indexers, the data gets indexed again.

What gives? Why is the event data missing when the Splunk Add-on for Infrastructure is deployed to my indexers?

[sloshburch · Accepted solution]

Short Answer

The Splunk Add-on for Infrastructure has configuration that is trying to convert the log event data into metrics data.

Explanation

The Splunk Add-on for Infrastructure is configured to convert any log event data with a source value of Perfmon:* to metrics data. Out of the box, Splunk's perfmon modular input will use Perfmon:* as the source value. So the Splunk Add-on for Infrastructure is trying to convert this data, even if you intended it to stay as log event data.

If you have Splunk Add-on for Microsoft Windows deployed, this should not be an issue because the default source value of the add-on's perfmon data is PerfmonMk:*. Many years ago, Splunk introduced the mode attribute with the value multikv. This value prints events formatted with multiple key-value pairs, not as single line events. This became the default in Splunk Add-on for Microsoft Windows.

Work Around 1: Upgrade Splunk Add-on for Microsoft Windows

If you upgrade Splunk Add-on for Microsoft Windows, the mode = multikv default will take affect and provide the same field extractions your used to.

Any custom knowledge objects that have the source hard coded as Perfmon:* would need to be adjusted to use PerfmonMk:*.

Work Around 2: Custom metrics source value

Follow the pattern of the Splunk Add-on for Infrastructure. Set a custom source value within the perfmon stanzas of inputs.conf and set the log-to-metric conversion to match that customer source value.

The source value can be hardcoded within the inputs.conf sent to each Windows endpoint that runs the perfmon input. For example:

[perfmon://CPU Load]
counters = % C1 Time;% C2 Time;% Idle Time;% Processor Time;% User Time;% Privileged Time;% Reserved Time;% Interrupt Time
instances = *
interval = 60
object = Processor
index = em_metrics
_meta =  env::lab entity_type::Windows_Hos
source = PerfmonMetric:CPU

Then deploy some related props.conf and transforms.conf in accordance to Where to install the App, Add-ons, and other dependencies for the Splunk® App for Infrastructure. You'll be creating a new custom stanza based on the source value used within the inputs.conf. That stanza's attributes can be a copy of the Splunk Add-on for Infrastructure equivalent. In our example:

[PerfmonMetric:CPU]
TRANSFORMS-_value_for_perfmon_metrics_store = value_for_perfmon_metrics_store
TRANSFORMS-metric_name_for_perfmon_metrics_store = metric_name_for_perfmon_metrics_store
TRANSFORMS-object_for_perfmon_metrics_store = object_for_perfmon_metrics_store
TRANSFORMS-instance_for_perfmon_metrics_store = instance_for_perfmon_metrics_store
TRANSFORMS-collection_for_perfmon_metrics_store = collection_for_perfmon_metrics_store
EVAL-metric_type = "gauge"
SEDCMD-remove-whitespace = s/ /_/g s/\s/ /g

Each TRANSFORMS attribute refers to a stanza within the Splunk Add-on for Infrastructure's transforms.conf. So make sure that file is co-located.