Splunk ITSI

Monitoring Disk Space in ITSI and having issue with Maintenance Windows

proyleJDS
Path Finder

Hi

I have a inherited a KPI that monitors disk space in ITSI, the search works fine and returns a results when the thresholds are breached however the episodes continue even when the server is in maintenance mode.

I think I know why but don't yet know how to work around it.

This is the KPI search:

| mstats avg(LogicalDisk.%_Free_Space) as "logicaldisk_free_space" avg(PhysicalDisk.%_Disk_Read_Time) as "physicaldisk_read_time" avg(PhysicalDisk.%_Disk_Write_Time) as "physicaldisk_write_time" avg(Network_Interface.Packets_Received/sec) as "network_packets_received_per_second" avg(Network_Interface.Packets_Sent/sec) as "network_packets_sent_per_second" avg(Network_Interface.Bytes_Received/sec) as "network_bytes_received_per_second" avg(Network_Interface.Bytes_Sent/sec) as "network_bytes_sent_per_second" avg(Network_Interface.Packets_Outbound_Errors) as "network_packets_outbound_errors" WHERE `sai_metrics_indexes` AND instance!=_Total instance!=P: by host,instance span=30s
| eval host_dev=host . ":" . instance
| eval "physicaldisk_total_time" = physicaldisk_read_time + physicaldisk_write_time
| eval "network_packets_total_per_second" = network_packets_received_per_second + network_packets_sent_per_second
| eval "network_mbs_total_per_second" = (network_bytes_received_per_second + network_bytes_sent_per_second)/1000000

 

The Threshold field is logical_free_space

The Split by field is host_dev which as you can see combines the host name with the disk device like this
HOST1234:C:

The data is filtered by service with the host field

The result in the service analyser looks good
image (1).png

Problem is with the Entity Name now being HOSTNAME:C: when the HOST is put into maintenance this KPI keeps creating episodes.

Can someone help me with a practical way to do this and still use maintenance mode successfully?

Cheers

Labels (2)
0 Karma
Get Updates on the Splunk Community!

Exporting Splunk Apps

Join us on Monday, October 21 at 11 am PT | 2 pm ET!With the app export functionality, app developers and ...

Cisco Use Cases, ITSI Best Practices, and More New Articles from Splunk Lantern

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Build Your First SPL2 App!

Watch the recording now!.Do you want to SPL™, too? SPL2, Splunk's next-generation data search and preparation ...