We are using ITSI to display Glass Table dashboards for staff to view the health of critical systems. We have thresholds set for Weekdays and Weekends. Obviously the thresholds for weekends are significantly lower than weekdays.
This creates a headache when a public holiday rolls around however, as we see weekend traffic volumes on a weekday, and so the dashboards begin to light up like a Christmas Tree. Hence we're forced to change the weekday thresholds to accept service levels as low as a weekend, which is far from ideal and hinders the tool from early detection and alerting.
Has anyone found a way to manage public holiday thresholds?
Have you tried using Maintenance Windows as a workaround ? Also you could try adjusting your thresholds using lookups for Holidays (we have solved it this way).
The problem with a maintenance window is that it hides the problem. It simply suppresses the alert and leaves everyone blind to service availability on what is still an important day of operation for the organisation (i.e. volumes may be lower, but criticality of services are just as high).
In a future release we hope to have a "Special Days" feature that takes care of thresholds for special days like Black Friday, Christmas, and other significant days for your organization. However, this functionality currently does not exist 😞
Keep your eye on the new features lists for each release and hopefully it'll be there soon: https://docs.splunk.com/Documentation/ITSI/latest/ReleaseNotes/Newfeatures
I've been trying out options as a workaround, the most suggested of which is the use of a lookup table. However the problem with the lookup is what to do when we know a search result falls on a public holiday. When a result is on a public holiday, you would assume we would exclude/ignore the event. This would produce a null result for the KPI. The biggest problem with this is that a lack of events returned is also a platform level indexing issues that are possible within Splunk... so public holidays and index performance issues would trigger the same KPI thresholds as each other (i.e. null = 0, and 0 equals bad).
Given ITSI is a tool aimed at service health monitoring, and that services are directly impacted by public holidays, it would seem this is critical functionality currently missing from the product.