topic Re: Perfmon data missing from two servers. in Getting Data In

Why is Perfmon data missing from two servers / Free disk space table?

thebankitgui — Thu, 26 Jan 2023 19:08:23 GMT

Good Morning,

I have been working on a task to gather the free disk space of servers we have Splunk Universal Forwarder on. I am down to getting data from all servers through the perfmon data. I have it for all servers but two. One of these is the Splunk deployment server (we're on Splunk Cloud). I have checked all the apps which might have inputs.conf with stanzas referring to "source="Perfmon:Free Disk Space" and I've looked in /etc/system/local on the Deployment Server. All the stanzas are at 0 and I've restarted Splunk after each change, I'm at a loss! Thank you in advance.

Scott

Re: Perfmon data missing from two servers.

gcusello — Wed, 25 Jan 2023 15:47:28 GMT

Hi @thebankitgui ,

perfmon is a windows counter, Splunk Cloud systems are Linux systems, so you cannot use the Splunk-TA-Windows to have this information.

You have to use the Splunk Add-On for Linux (https://splunkbase.splunk.com/app/833) enabling the free disk space counter.

But Anyway, free disk space on Splunk Cloud servers isn't your problem.

One additional information: using a Windows Deployment Server you'll encounter some issues related to the grants on scripts to execute on Linux servers, one of this is just the Free disk space script; for this reason isn't a best practice to use a Windows Deployment Server having target Linux servers.

Only for conclusion: I never saw a Splunk production system based on Windows, at best a test enviroment.

Ciao.

Giuseppe

Re: Perfmon data missing from two servers.

thebankitgui — Wed, 25 Jan 2023 15:55:41 GMT

Giuseppe,

Thank you for the response. There seems to be a misunderstanding. I have the free disk space for 9 of 11 servers that are all Windows Server 2019. The issue is that the data from 2 of 11 of the servers is not showing up, despite checking all the inputs.conf I can find. I was just thinking there had to be a file or setting I was missing or a Splunk behavior that would result in this behavior.

Scott

Re: Perfmon data missing from two servers.

gcusello — Wed, 25 Jan 2023 16:01:26 GMT

Hi @thebankitgui,

which Add-On are you using?

check if the user running Splunk on these two servers has the grants to execute the scripts and if there are differences with the other servers.

Ciao.

Giuseppe

Re: Perfmon data missing from two servers.

thebankitgui — Wed, 25 Jan 2023 17:21:11 GMT

Splunk_TA_windows and in the etc\deployment-apps\Splunk_TA_windows\local\inputs.conf I have the following:

## Logical Disk
[perfmon://LogicalDisk]
counters = % Free Space; Free Megabytes; Current Disk Queue Length; % Disk Time; Avg. Disk Queue Length; % Disk Read Time; Avg. Disk Read Queue Length; % Disk Write Time; Avg. Disk Write Queue Length; Avg. Disk sec/Transfer; Avg. Disk sec/Read; Avg. Disk sec/Write; Disk Transfers/sec; Disk Reads/sec; Disk Writes/sec; Disk Bytes/sec; Disk Read Bytes/sec; Disk Write Bytes/sec; Avg. Disk Bytes/Transfer; Avg. Disk Bytes/Read; Avg. Disk Bytes/Write; % Idle Time; Split IO/Sec
disabled = 0
instances = *
interval = 10
mode = single
object = LogicalDisk
useEnglishOnly=true
index=main

Re: Why is Perfmon data missing from two servers?

thebankitgui — Wed, 25 Jan 2023 20:24:03 GMT

I can't delete this thread but through trial and error I have gotten all servers to report free disk space, now to make it look pretty. Thank you for your help. 🙂

Re: Perfmon data missing from two servers.

gcusello — Thu, 26 Jan 2023 11:55:14 GMT

Hi @thebankitgui,

did you checked if the grants of the user sunning Splunk on these two servers are the same of the others?

I suppose that you're receiving other windows logs from these two servers.

Ciao.

Giuseppe

Re: Perfmon data missing from two servers.

thebankitgui — Thu, 26 Jan 2023 13:23:06 GMT

To be honest, part of my mistake might have been the "top 10 sources" when I have 12 hosts. Once I got the other two working, I kept only seeing 10. All set now. I just need to massage the data to appear how I like it. Ideally a clean date (YYYY-DD-MM), no time needed and % instead of the numbers after the decimal. Any SPL gurus that can help? Here's what I have, the table it produces is attached:

(index=main) sourcetype=perfmon:LogicalDisk instance!=_Total instance!=Harddisk* | eval FreePct-Other=case( match (instance, "C:"), null(), match(instance,"D:"), null(),true(),storage_free_percent), FreeMB-Other=case( match (instance, "C:"), null(), match(instance,"D:"), null(), true(),Free_Megabytes), FreePct-{instance}=storage_free_percent,FreeMB-{instance}=Free_Megabytes| search counter="% Free Space" | fields "_time", "host", "instance", "Value"

Re: Perfmon data missing from two servers.

gcusello — Thu, 26 Jan 2023 13:25:26 GMT

Hi @thebankitgui,

if one answer solves your need, please accept one answer for the other people of Community or tell me how I can help you.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

Re: Perfmon data missing from two servers.

thebankitgui — Thu, 26 Jan 2023 13:42:23 GMT

Giuseppe,

I appreciate your replies, I really do. Unfortunately I figured out the inputs.conf through trial and error, looking at indexes to see when the wrong one was set and adding a stanza in one case. Now that I've got that solved on my own, I put in some SPL that I could use help with optimizing but I can do another thread for that. It's all part of the same issue because the perfmon data is for free disk space.