I downvoted this post because it's disrespectful and/or inconsiderate in my eyes: the poster did not appear to read the question through. ... View more

Different thresholds (transcoders vs. SMTP relays vs. SQL servers), escalation paths (engineering, NOC, SOC?) , alert types (email, Slack), alert frequency for different hosts and entities depending on their purpose and criticality. Then it's: host up/down or in warning or critical service or application up/down - or in warning or critical (there can be multiple criteria for that depending on the application or service) high latencies CPU, disk, memory, network utilization - with varying thresholds depending on multiple factors ... and I am probably forgetting a thing or twenty. In Solarwinds, we have about 50-60 active alerts and they're a bear to manage due to no conditional execution in alert actions (e.g. tag based) - but at least they have escalations tiers - not something Splunk does. Going forward we need a better alert management system - in addition to fast alerts that don't bring the cluster down. ... View more

Thank you! Just to confirm: not even metrics would help? (Provided we're OK with waiting longer than a minute or two for the alert to fire.) Also: can you think of where this topic is discussed at length? (I can't be the 1st one to come up with such a question?) ... View more

If there is a possibility you didn't read the question fully or are attempting to answer a different one - please consider deleting or revising your answer. In my eyes, it doesn't even attempt to answer the question - which I believe to be inconsiderate, at the very least - and perhaps the reason for the "tone". Cheers. ... View more

The question is specific - or at least contains very specific sub-questions. Specific scenarios knocking out our 3-strong cluster are: - 5+ realtime searches on fields extracted at index time, - Someone else in my team converting a dashboard with expensive searches to run in realtime (they are used to that in other tools like Solarwinds and Datadog, and when I tell them Splunk is not like that, they stop using it). ... however those scenarios were omitted on purpose, to keep the focus laser sharp: in an environment where fast alerts on a large number of metrics is important, is Splunk the right product for the task? If so, what are the mechanisms to accomplish that? (Because OOTB, Splunk is not suitable for it - at least not in my experience.) So does your answer imply "no such OOTB mechanism"? (Outside of a rather steep learning curve - or hiring a Splunk specialist - to optimize things? And, of course, metrics? Or are metrics not the right mechanism?) What is the appropriate mechanism in Splunk to have multiple (potentially hundreds) of alerts that are based on the latest events, rather than real-time or timeframe searches, while keeping our Splunk deployment sane and simple? Now to your answer... There is no need to have individual alerts for different volumes, NAS volumes, SAN volumes, etc. - merge them all into one. There isn't? Well then. Is there a sample or a template search and alert available in official Splunk documentation that I could easily integrate into my environment? (Focus on "easily" - so that someone else in my team not intimately familiar with Splunk could do that?) No such template? Rethink your answer then? How does one merge multiple alerts into one if they need to have different alert actions? E.g. alert different teams based on the host, volume, severity, threshold, escalation level, etc.? How does one merge alerts when the underlying searches - and resulting alerts are very different? (Local volumes are searched across all hosts excluding certain volume types while SAN and NAS volumes - at least with respect to "disk full" alerts - are only searched on specific hosts? If merging them makes the search unwieldy, too complex, hard to manage - rethink your answer? It seems your answer implies hiring someone with 3+ (or is it 10+?) years of Splunk experience who could optimize the searches alerts sufficiently to make them more performant? If this sound about right - perhaps rethink your answer? ... View more

Good. Delete your answer then? ... View more

You might not like my real answer... Deflection. Go easy. Don't use splunk as a monitoring tool. Now that may be an answer to one my questions: Is Splunk not the right product for the task? ...although it deflects, again: says "don't use" rather than, "no, it is not the right product". See the difference? Besides, I suspect Metrics and Convert event logs to metric data points are either better answers - or at least point me in the right direction. You're correct, I do not like your answer - and not because it steps on my toes or contradicts by biases. It's because it contains factually incorrect information (about Solarwinds), and doesn't even attempt to answer the actual question. ... View more

Sounds like the answer is a "no"? Should have been a comment though, not an answer - I don't see even a half-serious attempt to answer the actual question. ... View more

Expanding the timeframe of such a search increases its "cost" - its time to run. I.e. Splunk does not stop searching when it finds the most recent event - it keeps going through all of them. This doesn't feel efficient to me. Why this matters: if the task is get the value of a field in the last event in a search no matter when that last event happened (30 seconds ago? Two years?) - one should search "all time", and this will take a long, long time to complete across a decent size dataset. If on the other hand stats first could stop searching once it found the last event - that would dramatically decrease the cost. ... View more

Thank you - this works, and I learned a couple of things! P.S. A simplified version of your answer: Base Search: sourcetype="tomcat:vantage" | eval "Field Extraction Error(s)" = if(isnull(message),"Present","Not Present") | stats sparkline count by "Field Extraction Error(s)" | addcoltotals labelfield="Field Extraction Error(s)" label="Total Events" Post-process searches: | search "Field Extraction Error(s)" IN ("Present") | table count | table "Field Extraction Error(s)" sparkline count | search "Field Extraction Error(s)" IN ("Total Events") | table count | search "Field Extraction Error(s)" IN ("Not Present") | table count ... View more

Put the master in maintenance, rolling reboot of all indexers; checking bucket status between reboots to ensure integrity. That said - I can't remember now if I also tried manual rolling service restarts. (Automatic rolling restart from the master wasn't available due to that degraded index.) ... View more