Splunk ITSI

Monitoring Disk Space in ITSI and having issue with Maintenance Windows

proyleJDS
Path Finder

Hi

I have a inherited a KPI that monitors disk space in ITSI, the search works fine and returns a results when the thresholds are breached however the episodes continue even when the server is in maintenance mode.

I think I know why but don't yet know how to work around it.

This is the KPI search:

| mstats avg(LogicalDisk.%_Free_Space) as "logicaldisk_free_space" avg(PhysicalDisk.%_Disk_Read_Time) as "physicaldisk_read_time" avg(PhysicalDisk.%_Disk_Write_Time) as "physicaldisk_write_time" avg(Network_Interface.Packets_Received/sec) as "network_packets_received_per_second" avg(Network_Interface.Packets_Sent/sec) as "network_packets_sent_per_second" avg(Network_Interface.Bytes_Received/sec) as "network_bytes_received_per_second" avg(Network_Interface.Bytes_Sent/sec) as "network_bytes_sent_per_second" avg(Network_Interface.Packets_Outbound_Errors) as "network_packets_outbound_errors" WHERE `sai_metrics_indexes` AND instance!=_Total instance!=P: by host,instance span=30s
| eval host_dev=host . ":" . instance
| eval "physicaldisk_total_time" = physicaldisk_read_time + physicaldisk_write_time
| eval "network_packets_total_per_second" = network_packets_received_per_second + network_packets_sent_per_second
| eval "network_mbs_total_per_second" = (network_bytes_received_per_second + network_bytes_sent_per_second)/1000000

 

The Threshold field is logical_free_space

The Split by field is host_dev which as you can see combines the host name with the disk device like this
HOST1234:C:

The data is filtered by service with the host field

The result in the service analyser looks good
image (1).png

Problem is with the Entity Name now being HOSTNAME:C: when the HOST is put into maintenance this KPI keeps creating episodes.

Can someone help me with a practical way to do this and still use maintenance mode successfully?

Cheers

Labels (2)
0 Karma
Get Updates on the Splunk Community!

AppDynamics Summer Webinars

This summer, our mighty AppDynamics team is cooking up some delicious content on YouTube Live to satiate your ...

SOCin’ it to you at Splunk University

Splunk University is expanding its instructor-led learning portfolio with dedicated Security tracks at .conf25 ...

Credit Card Data Protection & PCI Compliance with Splunk Edge Processor

Organizations handling credit card transactions know that PCI DSS compliance is both critical and complex. The ...