Splunk ITSI

Monitoring Disk Space in ITSI and having issue with Maintenance Windows

proyleJDS
Path Finder

Hi

I have a inherited a KPI that monitors disk space in ITSI, the search works fine and returns a results when the thresholds are breached however the episodes continue even when the server is in maintenance mode.

I think I know why but don't yet know how to work around it.

This is the KPI search:

| mstats avg(LogicalDisk.%_Free_Space) as "logicaldisk_free_space" avg(PhysicalDisk.%_Disk_Read_Time) as "physicaldisk_read_time" avg(PhysicalDisk.%_Disk_Write_Time) as "physicaldisk_write_time" avg(Network_Interface.Packets_Received/sec) as "network_packets_received_per_second" avg(Network_Interface.Packets_Sent/sec) as "network_packets_sent_per_second" avg(Network_Interface.Bytes_Received/sec) as "network_bytes_received_per_second" avg(Network_Interface.Bytes_Sent/sec) as "network_bytes_sent_per_second" avg(Network_Interface.Packets_Outbound_Errors) as "network_packets_outbound_errors" WHERE `sai_metrics_indexes` AND instance!=_Total instance!=P: by host,instance span=30s
| eval host_dev=host . ":" . instance
| eval "physicaldisk_total_time" = physicaldisk_read_time + physicaldisk_write_time
| eval "network_packets_total_per_second" = network_packets_received_per_second + network_packets_sent_per_second
| eval "network_mbs_total_per_second" = (network_bytes_received_per_second + network_bytes_sent_per_second)/1000000

 

The Threshold field is logical_free_space

The Split by field is host_dev which as you can see combines the host name with the disk device like this
HOST1234:C:

The data is filtered by service with the host field

The result in the service analyser looks good
image (1).png

Problem is with the Entity Name now being HOSTNAME:C: when the HOST is put into maintenance this KPI keeps creating episodes.

Can someone help me with a practical way to do this and still use maintenance mode successfully?

Cheers

Labels (2)
0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Leveraging Automated Threat Analysis Across the Splunk Ecosystem

Are you leveraging automation to its fullest potential in your threat detection strategy?Our upcoming Security ...

Can’t Make It to Boston? Stream .conf25 and Learn with Haya Husain

Boston may be buzzing this September with Splunk University and .conf25, but you don’t have to pack a bag to ...

Splunk Lantern’s Guide to The Most Popular .conf25 Sessions

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...