I've been collecting data using Splunk's perfmon
modular input as per Splunk Add-on for Microsoft Windows. Yet after I deployed the Splunk Add-on for Infrastructure, I no longer see any event data.
I am confident it is related to the Splunk Add-on for Infrastructure because when I remove that add-on from my indexers, the data gets indexed again.
What gives? Why is the event data missing when the Splunk Add-on for Infrastructure is deployed to my indexers?
The Splunk Add-on for Infrastructure has configuration that is trying to convert the log event data into metrics data.
The Splunk Add-on for Infrastructure is configured to convert any log event data with a source value of Perfmon:*
to metrics data. Out of the box, Splunk's perfmon modular input will use Perfmon:*
as the source value. So the Splunk Add-on for Infrastructure is trying to convert this data, even if you intended it to stay as log event data.
If you have Splunk Add-on for Microsoft Windows deployed, this should not be an issue because the default source value of the add-on's perfmon data is PerfmonMk:*
. Many years ago, Splunk introduced the mode
attribute with the value multikv
. This value prints events formatted with multiple key-value pairs, not as single line events. This became the default in Splunk Add-on for Microsoft Windows.
If you upgrade Splunk Add-on for Microsoft Windows, the mode = multikv
default will take affect and provide the same field extractions your used to.
Any custom knowledge objects that have the source hard coded as Perfmon:*
would need to be adjusted to use PerfmonMk:*
.
Follow the pattern of the Splunk Add-on for Infrastructure. Set a custom source value within the perfmon
stanzas of inputs.conf
and set the log-to-metric conversion to match that customer source value.
The source value can be hardcoded within the inputs.conf
sent to each Windows endpoint that runs the perfmon input. For example:
[perfmon://CPU Load]
counters = % C1 Time;% C2 Time;% Idle Time;% Processor Time;% User Time;% Privileged Time;% Reserved Time;% Interrupt Time
instances = *
interval = 60
object = Processor
index = em_metrics
_meta = env::lab entity_type::Windows_Hos
source = PerfmonMetric:CPU
Then deploy some related props.conf
and transforms.conf
in accordance to Where to install the App, Add-ons, and other dependencies for the Splunk® App for Infrastructure. You'll be creating a new custom stanza based on the source value used within the inputs.conf
. That stanza's attributes can be a copy of the Splunk Add-on for Infrastructure equivalent. In our example:
[PerfmonMetric:CPU]
TRANSFORMS-_value_for_perfmon_metrics_store = value_for_perfmon_metrics_store
TRANSFORMS-metric_name_for_perfmon_metrics_store = metric_name_for_perfmon_metrics_store
TRANSFORMS-object_for_perfmon_metrics_store = object_for_perfmon_metrics_store
TRANSFORMS-instance_for_perfmon_metrics_store = instance_for_perfmon_metrics_store
TRANSFORMS-collection_for_perfmon_metrics_store = collection_for_perfmon_metrics_store
EVAL-metric_type = "gauge"
SEDCMD-remove-whitespace = s/ /_/g s/\s/ /g
Each TRANSFORMS
attribute refers to a stanza within the Splunk Add-on for Infrastructure's transforms.conf
. So make sure that file is co-located.
The Splunk Add-on for Infrastructure has configuration that is trying to convert the log event data into metrics data.
The Splunk Add-on for Infrastructure is configured to convert any log event data with a source value of Perfmon:*
to metrics data. Out of the box, Splunk's perfmon modular input will use Perfmon:*
as the source value. So the Splunk Add-on for Infrastructure is trying to convert this data, even if you intended it to stay as log event data.
If you have Splunk Add-on for Microsoft Windows deployed, this should not be an issue because the default source value of the add-on's perfmon data is PerfmonMk:*
. Many years ago, Splunk introduced the mode
attribute with the value multikv
. This value prints events formatted with multiple key-value pairs, not as single line events. This became the default in Splunk Add-on for Microsoft Windows.
If you upgrade Splunk Add-on for Microsoft Windows, the mode = multikv
default will take affect and provide the same field extractions your used to.
Any custom knowledge objects that have the source hard coded as Perfmon:*
would need to be adjusted to use PerfmonMk:*
.
Follow the pattern of the Splunk Add-on for Infrastructure. Set a custom source value within the perfmon
stanzas of inputs.conf
and set the log-to-metric conversion to match that customer source value.
The source value can be hardcoded within the inputs.conf
sent to each Windows endpoint that runs the perfmon input. For example:
[perfmon://CPU Load]
counters = % C1 Time;% C2 Time;% Idle Time;% Processor Time;% User Time;% Privileged Time;% Reserved Time;% Interrupt Time
instances = *
interval = 60
object = Processor
index = em_metrics
_meta = env::lab entity_type::Windows_Hos
source = PerfmonMetric:CPU
Then deploy some related props.conf
and transforms.conf
in accordance to Where to install the App, Add-ons, and other dependencies for the Splunk® App for Infrastructure. You'll be creating a new custom stanza based on the source value used within the inputs.conf
. That stanza's attributes can be a copy of the Splunk Add-on for Infrastructure equivalent. In our example:
[PerfmonMetric:CPU]
TRANSFORMS-_value_for_perfmon_metrics_store = value_for_perfmon_metrics_store
TRANSFORMS-metric_name_for_perfmon_metrics_store = metric_name_for_perfmon_metrics_store
TRANSFORMS-object_for_perfmon_metrics_store = object_for_perfmon_metrics_store
TRANSFORMS-instance_for_perfmon_metrics_store = instance_for_perfmon_metrics_store
TRANSFORMS-collection_for_perfmon_metrics_store = collection_for_perfmon_metrics_store
EVAL-metric_type = "gauge"
SEDCMD-remove-whitespace = s/ /_/g s/\s/ /g
Each TRANSFORMS
attribute refers to a stanza within the Splunk Add-on for Infrastructure's transforms.conf
. So make sure that file is co-located.