Alerting

How to select a partition in the DMC Alert - Near Critical Disk Usage?

btmoises
Loves-to-Learn

Hello

I'm working on the setup of the alert when the disk space usage reaches above 80.

However, I don't how to change in the query the partition that we need to check.

The main partition is installed the Splunk service, however, i want to set the alert for another partition, the one that stores the logs.

Here is the search for the alarm:

| rest splunk_server_group=dmc_group_* /services/server/status/partitions-space
| eval free = if(isnotnull(available), available, free)
| eval usage = capacity - free
| eval pct_usage = floor(usage / capacity * 100)
| where pct_usage > 30
| stats first(fs_type) as fs_type first(capacity) AS capacity first(usage) AS usage first(pct_usage) AS pct_usage by splunk_server, mount_point
| eval usage = round(usage / 1024, 2)
| eval capacity = round(capacity / 1024, 2)
| rename splunk_server AS Instance mount_point as "Mount Point", fs_type as "File System Type", usage as "Usage (GB)", capacity as "Capacity (GB)", pct_usage as "Usage (%)"

And the result for the search:

 

btmoises_0-1665640761149.png

And the partition that we need to monitoring is the next one:

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg00-root 1014M 84M 931M 9% /
/dev/mapper/vg00-usr 4.0G 1.8G 2.3G 45% /usr
/dev/sda1 1014M 192M 823M 19% /boot
/dev/mapper/vg00-opt 10G 6.3G 3.8G 63% /opt
/dev/mapper/vg01-splunk 32G 15G 18G 47% /var/log/splunk

How can I change the query, so the search is done on the last partition?

Regards!

 

Labels (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @btmoises,

this is a Linux server, so probably you installed on this server also the Splunk_TA-nix.

In this case, you already have the information about all the partitions of your system and you could run a search like the following:

index=os  sourcetype=df host=$host$ 
| head 1
| multikv 
| table Filesystem Type Size Used Avail UsePct MountedOn

to display the status of all partitions, then you can create your alert for the partition you want:

index=os  sourcetype=df host=$host$ 
| head 1
| multikv 
| table Filesystem Type Size Used Avail UsePct MountedOn
| search FileSystem="/var/log/splunk" UsePct="80%"

Ciao.

Giuseppe

0 Karma

btmoises
Loves-to-Learn

Hello @gcusello 

I tried the search that you suggested but I didn't get the results

btmoises_0-1665716938192.png

btmoises_1-1665717024432.png

 

btmoises_2-1665717100615.png

btmoises_3-1665717138102.png

 I did something wrong?

Thank you!

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @btmoises,

$host$ is the value of a dropdown list to choose the host to monitor, you have to use the hostname of the host to monitor:

index=os  sourcetype=df host=<your_host_to_monitor> 
| head 1
| multikv 
| table Filesystem Type Size Used Avail UsePct MountedOn

Ciao.

Giuseppe

0 Karma

btmoises
Loves-to-Learn

Hello 

It seems that the index=os shows no data, the same with the host and sourcetype.

btmoises_0-1665807440237.png

 

I use index="_*" to check all the data sources, above you can see all the sources.

So, the /dev/mapper/vg01-splunk is the $host$ right?

Thank you

 

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @btmoises,

if you installed the Splunk_TA-nix, see where you archive data, usually index=os, so use this index in the search.

If you haven't this TA, install it or find a different way to extract the disk space value.

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...