Splunk IT Service Intelligence

Monitoring Disk Space in ITSI and having issue with Maintenance Windows

proyleJDS
Explorer

Hi

I have a inherited a KPI that monitors disk space in ITSI, the search works fine and returns a results when the thresholds are breached however the episodes continue even when the server is in maintenance mode.

I think I know why but don't yet know how to work around it.

This is the KPI search:

| mstats avg(LogicalDisk.%_Free_Space) as "logicaldisk_free_space" avg(PhysicalDisk.%_Disk_Read_Time) as "physicaldisk_read_time" avg(PhysicalDisk.%_Disk_Write_Time) as "physicaldisk_write_time" avg(Network_Interface.Packets_Received/sec) as "network_packets_received_per_second" avg(Network_Interface.Packets_Sent/sec) as "network_packets_sent_per_second" avg(Network_Interface.Bytes_Received/sec) as "network_bytes_received_per_second" avg(Network_Interface.Bytes_Sent/sec) as "network_bytes_sent_per_second" avg(Network_Interface.Packets_Outbound_Errors) as "network_packets_outbound_errors" WHERE `sai_metrics_indexes` AND instance!=_Total instance!=P: by host,instance span=30s
| eval host_dev=host . ":" . instance
| eval "physicaldisk_total_time" = physicaldisk_read_time + physicaldisk_write_time
| eval "network_packets_total_per_second" = network_packets_received_per_second + network_packets_sent_per_second
| eval "network_mbs_total_per_second" = (network_bytes_received_per_second + network_bytes_sent_per_second)/1000000

 

The Threshold field is logical_free_space

The Split by field is host_dev which as you can see combines the host name with the disk device like this
HOST1234:C:

The data is filtered by service with the host field

The result in the service analyser looks good
image (1).png

Problem is with the Entity Name now being HOSTNAME:C: when the HOST is put into maintenance this KPI keeps creating episodes.

Can someone help me with a practical way to do this and still use maintenance mode successfully?

Cheers

Labels (2)
0 Karma
Get Updates on the Splunk Community!

Combine Multiline Logs into a Single Event with SOCK - a Guide for Advanced Users

This article is the continuation of the “Combine multiline logs into a single event with SOCK - a step-by-step ...

Everything Community at .conf24!

You may have seen mention of the .conf Community Zone 'round these parts and found yourself wondering what ...

Index This | I’m short for "configuration file.” What am I?

May 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with a Special ...