Getting Data In

Why are none of the Perfmon inputs are sending Data Into Metrics Index and not showing errors?

Razziq
Explorer

Hello,

We're running into an issue with a UF sending data to a new metrics index under an app deployed by our deployment server. None of the perfmon inputs are sending data into our new index, and we're not seeing any errors.

We also have the Splunk TA Windows Base app deployed to these same servers, and if we test adjusting the inputs.conf stanzas in the TA app to send perfmon metrics to our new index, it works fine.

Below are the input stanzas from both apps:


Custom app:

## Process
[perfmon://Process]
counters = % Processor Time; % User Time- Private
disabled = 0
instances = *
interval = 60
mode = single
object = Process
useEnglishOnly=true
index = custom_metrics


TA Windows Base originally:

## Process
[perfmon://Process]
counters = % Processor Time; % User Time; % Privileged Time; Virtual Bytes Peak; Virtual Bytes; Page Faults/sec; Working Set Peak; Working Set; Page File Bytes Peak; Page File Bytes; Private Bytes; Thread Count; Priority Base; Elapsed Time; ID Process; Creating Process ID; Pool Paged Bytes; Pool Nonpaged Bytes; Handle Count; IO Read Operations/sec; IO Write Operations/sec; IO Data Operations/sec; IO Other Operations/sec; IO Read Bytes/sec; IO Write Bytes/sec; IO Data Bytes/sec; IO Other Bytes/sec; Working Set - Private
disabled = 0
instances = *
interval = 10
mode = single
object = Process
useEnglishOnly=true
index = ava_cs_metrics

 

TA Windows Base changed to point to our new index (which writes to the index fine) :

## Process
[perfmon://Process]
counters = % Processor Time; % User Time; % Privileged Time; Virtual Bytes Peak; Virtual Bytes; Page Faults/sec; Working Set Peak; Working Set; Page File Bytes Peak; Page File Bytes; Private Bytes; Thread Count; Priority Base; Elapsed Time; ID Process; Creating Process ID; Pool Paged Bytes; Pool Nonpaged Bytes; Handle Count; IO Read Operations/sec; IO Write Operations/sec; IO Data Operations/sec; IO Other Operations/sec; IO Read Bytes/sec; IO Write Bytes/sec; IO Data Bytes/sec; IO Other Bytes/sec; Working Set - Private
disabled = 0
instances = *
interval = 10
mode = single
object = Process
useEnglishOnly=true
index = custom_metrics

 

Labels (2)
0 Karma

VatsalJagani
Champion

Can you please post any errors if you are seeing in Splunk's internal logs related to these inputs?

0 Karma

Razziq
Explorer

VatsalJagani,

We do not see any errors in any of the UF logs, in the Splunk interface or searching _* indexes. Is there anywhere else we should check? It's like the data isn't even trying to write to the metrics index from the custom app, whereas from TA Windows Base, it works fine (though we cannot permanently change it here, as TA is deployed to all servers via the deployment app)

0 Karma

VatsalJagani
Champion

If the same configuration with just a normal (raw) index is working fine then make sure you have a metric index created (custom_metric) on your indexer.

You can do a test configuration on a test machine first if you cannot make modifications on servers.

0 Karma

Razziq
Explorer

We do have a custom metrics index created and the Windows Base TA can write to it. The custom app cannot and I wasn't sure if the Windows Base TA had a config that blocked others on the same host from writing to metrics indexes.

0 Karma

VatsalJagani
Champion

Both metric and event will not work together as Splunk look at the stanza name only to see whether it's input or not.

So you are updating the attributes of the same input regardless of whether they are disabled or not.

So you need to remove the index attribute from Windows TA in order to make your custom App's index in priority.

 

For example:

# myApp
[perfmon://Process]
counters = % Processor Time; % User Time- Private
disabled = 0
instances = *
interval = 60
mode = single
object = Process
useEnglishOnly=true
index = custom_metrics


## Splunk_TA_windows
[perfmon://Process]
counters = % Processor Time; % User Time; % Privileged Time; Virtual Bytes Peak; Virtual Bytes; Page Faults/sec; Working Set Peak; Working Set; Page File Bytes Peak; Page File Bytes; Private Bytes; Thread Count; Priority Base; Elapsed Time; ID Process; Creating Process ID; Pool Paged Bytes; Pool Nonpaged Bytes; Handle Count; IO Read Operations/sec; IO Write Operations/sec; IO Data Operations/sec; IO Other Operations/sec; IO Read Bytes/sec; IO Write Bytes/sec; IO Data Bytes/sec; IO Other Bytes/sec; Working Set - Private
disabled = 0
instances = *
interval = 10
mode = single
object = Process
useEnglishOnly=true
index = ava_cs_metrics

 

(Note - capital S has higher precedence than small m in the App name as they follow ASCII sequence)

What Splunk sees:

[perfmon://Process]
counters = % Processor Time; % User Time; % Privileged Time; Virtual Bytes Peak; Virtual Bytes; Page Faults/sec; Working Set Peak; Working Set; Page File Bytes Peak; Page File Bytes; Private Bytes; Thread Count; Priority Base; Elapsed Time; ID Process; Creating Process ID; Pool Paged Bytes; Pool Nonpaged Bytes; Handle Count; IO Read Operations/sec; IO Write Operations/sec; IO Data Operations/sec; IO Other Operations/sec; IO Read Bytes/sec; IO Write Bytes/sec; IO Data Bytes/sec; IO Other Bytes/sec; Working Set - Private
disabled = 0
instances = *
interval = 10
mode = single
object = Process
useEnglishOnly=true
index = ava_cs_metrics

 

  • This is true even when you disable the input on Splunk_TA_windows, as Splunk is looking at all attributed from all Apps for the same stanza and deciding.
  • If the attribute is common, the one coming from higher precedence App takes priority.

 

I hope this helps!!!

0 Karma

Razziq
Explorer

Thanks for the information! I did a little more testing by adding ## A_Custom in the inputs for the custom app and leaving the TA stanza in place, but this didn't work.

I also completely removed the [Perfmon://Process] from the other conf files on the host (in local and default for the TA apps) and restarted the UF, but we're still not seeing any metrics write to the index from the custom app. I ran btool on the UF server and I only see the custom app's inputs for [Permon://Process].

D:\SplunkUniversalForwarder\etc\apps\custom_app\local\inputs.conf index = customer_metrics
D:\SplunkUniversalForwarder\etc\apps\custom_app\local\inputs.conf interval = 60
D:\SplunkUniversalForwarder\etc\apps\custom_app\local\inputs.conf mode = single
D:\SplunkUniversalForwarder\etc\apps\custom_app\local\inputs.conf object = Memory
D:\SplunkUniversalForwarder\etc\apps\custom_app\local\inputs.conf sourcetype = PerfmonMetrics:Memory
D:\SplunkUniversalForwarder\etc\apps\custom_app\local\inputs.conf useEnglishOnly = true
D:\SplunkUniversalForwarder\etc\apps\custom_app\local\inputs.conf [Perfmon://Process]
D:\SplunkUniversalForwarder\etc\apps\custom_app\local\inputs.conf counters = % Processor Time; % User Time; % Privileged Time; Virtual Bytes Peak; Virtual Bytes; Page Faults/sec; Working Set Peak; Working Set; Page File Bytes Peak; Page File Bytes; Private Bytes; Thread Count; Priority Base; Elapsed Time; ID Process; Creating Process ID; Pool Paged Bytes; Pool Nonpaged Bytes; Handle Count; IO Read Operations/sec; IO Write Operations/sec; IO Data Operations/sec; IO Other Operations/sec; IO Read Bytes/sec; IO Write Bytes/sec; IO Data Bytes/sec; IO Other Bytes/sec; Working Set - Private
D:\SplunkUniversalForwarder\etc\apps\custom_app\local\inputs.conf disabled = 0

 

0 Karma
Get Updates on the Splunk Community!

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...