The question is specific - or at least contains very specific sub-questions. Specific scenarios knocking out our 3-strong cluster are:
- 5+ realtime searches on fields extracted at index time,
- Someone else in my team converting a dashboard with expensive searches to run in realtime (they are used to that in other tools like Solarwinds and Datadog, and when I tell them Splunk is not like that, they stop using it).
... however those scenarios were omitted on purpose, to keep the focus laser sharp: in an environment where fast alerts on a large number of metrics is important, is Splunk the right product for the task? If so, what are the mechanisms to accomplish that? (Because OOTB, Splunk is not suitable for it - at least not in my experience.)
So does your answer imply "no such OOTB mechanism"? (Outside of a rather steep learning curve - or hiring a Splunk specialist - to optimize things? And, of course, metrics? Or are metrics not the right mechanism?)
What is the appropriate mechanism in Splunk to have multiple (potentially hundreds) of alerts that are based on the latest events, rather than real-time or timeframe searches, while keeping our Splunk deployment sane and simple?
Now to your answer...
There is no need to have individual alerts for different volumes, NAS volumes, SAN volumes, etc. - merge them all into one.
There isn't? Well then.
Is there a sample or a template search and alert available in official Splunk documentation that I could easily integrate into my environment? (Focus on "easily" - so that someone else in my team not intimately familiar with Splunk could do that?) No such template? Rethink your answer then?
How does one merge multiple alerts into one if they need to have different alert actions? E.g. alert different teams based on the host, volume, severity, threshold, escalation level, etc.?
How does one merge alerts when the underlying searches - and resulting alerts are very different? (Local volumes are searched across all hosts excluding certain volume types while SAN and NAS volumes - at least with respect to "disk full" alerts - are only searched on specific hosts? If merging them makes the search unwieldy, too complex, hard to manage - rethink your answer?
It seems your answer implies hiring someone with 3+ (or is it 10+?) years of Splunk experience who could optimize the searches alerts sufficiently to make them more performant? If this sound about right - perhaps rethink your answer?
... View more