All Apps and Add-ons

Filtering out nfs-mounted filesystems for DF_STORAGE data

Ranazar
Path Finder

I noticed that the DF_STORAGE data from nmon (eventtype="nmon:performance" type="DF_STORAGE") includes stats for NFS-mounted filesystems as well - and they also show up in the datamodel NMON_Data_JFS (where nodename=JFS.DF_STORAGE).

Right now I'm just filtering these out by specifying filesystem="/dev*" - is there a better way to exclude them?

I'm using v1.3.31 of the addon on the servers themselves, and v1.9.15 of the app (community edition)..

0 Karma
1 Solution

guilmxm
Influencer

Hi @Ranazar

The DF_STORAGE data collection is driven by an external data collection documented here:

https://ta-nmon.readthedocs.io/en/latest/external.html

In a few lines, this is managed by the following script:

TA-nmon/TA-nmon/bin/nmon_external_cmd/nmon_external_snap.sh

And specially this variable being set at the script execution:

DF_TABLE=`df -k -P | sed '1d' | egrep -v '\/proc$|/dev$|\/run$|^tmpfs.*\/dev.*$|^tmpfs.*\/run.*$|^tmpfs.*\/sys.*$|^tmpfs.*\/var.*$' | awk '{print $6}'`

If you run this command on one of the boxes:

df -k -P | sed '1d' | egrep -v '\/proc$|/dev$|\/run$|^tmpfs.*\/dev.*$|^tmpfs.*\/run.*$|^tmpfs.*\/sys.*$|^tmpfs.*\/var.*$'

This would provide the list of the file-systems that will be taken in charge by the data collection.

So you could update this to exclude additional file-systems, if you modify the script and push the update, this will be taken in charge at the next life cycle of the nmon binary (max 24h per server)

Can you show what the file-system looks like ?
It might makes sense for me to update this to exclude NFS mounted file-systems.

Guilhem

View solution in original post

0 Karma

guilmxm
Influencer

Hi @Ranazar

The DF_STORAGE data collection is driven by an external data collection documented here:

https://ta-nmon.readthedocs.io/en/latest/external.html

In a few lines, this is managed by the following script:

TA-nmon/TA-nmon/bin/nmon_external_cmd/nmon_external_snap.sh

And specially this variable being set at the script execution:

DF_TABLE=`df -k -P | sed '1d' | egrep -v '\/proc$|/dev$|\/run$|^tmpfs.*\/dev.*$|^tmpfs.*\/run.*$|^tmpfs.*\/sys.*$|^tmpfs.*\/var.*$' | awk '{print $6}'`

If you run this command on one of the boxes:

df -k -P | sed '1d' | egrep -v '\/proc$|/dev$|\/run$|^tmpfs.*\/dev.*$|^tmpfs.*\/run.*$|^tmpfs.*\/sys.*$|^tmpfs.*\/var.*$'

This would provide the list of the file-systems that will be taken in charge by the data collection.

So you could update this to exclude additional file-systems, if you modify the script and push the update, this will be taken in charge at the next life cycle of the nmon binary (max 24h per server)

Can you show what the file-system looks like ?
It might makes sense for me to update this to exclude NFS mounted file-systems.

Guilhem

0 Karma

Ranazar
Path Finder

Perfect, thanks! That gives me a good option to filter it out at collection type.

The filter is a bit of an interesting challenge. I only caught this because one of the common mount points - /mnt - showed up in the data, but we've got NFS filesystems on various mount points other than /mnt.

It looks like the filesystems we'd want to include always start with /dev, so a piped command like | egrep '^\/dev' would catch them all. I took a look at a Linux server (we don't have that many), and it looks like the ones which don't start with /dev are already being filtered out by your egrep -v:

$ df -k -P
Filesystem 1024-blocks Used Available Capacity Mounted on
/dev/mapper/root xxx xxx xxx 32% /
devtmpfs xxx xxx xxx 0% /dev
tmpfs xxx xxx xxx 0% /dev/shm
tmpfs xxx xxx xxx 3% /run
tmpfs xxx 0 xxx 0% /sys/fs/cgroup
/dev/mapper/lvol0 xxx xxx xxx 19% /fs1
/dev/sda1 xxx xxx xxx 26% /boot
tmpfs xxx 0 xxx 0% /run/user/1417441623
tmpfs xxx 0 xxx 0% /run/user/995`

Alternatively, I see a -T local flag in the AIX df command that takes care of it nicely; I'm not familiar enough with the Linux version though (maybe -x).

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...