topic Re: Splunk Alerts & Dashboard Panels in Splunk Search

Splunk Alerts & Dashboard Panels

craigwilkinson — Tue, 11 Apr 2017 05:50:07 GMT

Hi All,

I have configured an alert to trigger based on when a the tcpout queue size breaches 80% - as per the SPL below:

earliest=-10m  index=_internal host=*hfw* source=*metrics.log group=queue name=tcpout* 
| eval queuecapacity_percent=round((current_size/max_size)*100,2), eval current_size_mb=round((current_size/1024/1024),2), eval max_size_mb=round((max_size/1024/1024),2) 
| where queuecapacity_percent >= 80 
| fields host, index, name, group, current_size, largest_size, max_size, date_year, date_month, date_mday, date_hour, date_minute, date_second, queuecapacity_percent, current_size_mb, max_size_mb

In addition to this, I'd like to have a Single Value panel on a dashboard, which will display the current TCP Output queue size - which I have written so far with:

earliest=-10m  index=_internal host=*hfw* source=*metrics.log group=queue name=tcpout* | eval queuecapacity_percent=round((current_size/max_size)*100,2) | eval current_size_mb=round((current_size/1024/1024),2) | eval max_size_mb=round((max_size/1024/1024),2)  | timechart span=1m max(queuecapacity_percent)

My Question is: What are the best practices when creating alerts, and dashboard panels?
These two objects are looking at the same data, however I want an alert & email to trigger when the threshold breaches 80%.

Is it common to have a search dedicated to the alert, and one dedicated to the dashboard panel?
How can I combine these into a single splunk object to save on performance?
Is there are better way to approach this?

Any help is greatly appreciated!
Apologies if this question has been asked before.

Thanks,

Craig

Re: Splunk Alerts & Dashboard Panels

niketn — Tue, 11 Apr 2017 07:26:26 GMT

@craigwilkinson... Ideally Dashboards are created so that Users can get insight to what is happening in their system at any given point of time... Historical, Real Time or Predictive. If they are setup on a monitoring screen they might either run a real-time search (based on Splunk infrastructure) or else refresh periodically.

Once SLA missed or KPI breach situation triggers an alert like Queue capacity above 80% etc, Users can resort to such dashboards for correlating and investigating further (even without having knowledge of Splunk or Underlying data). For situations like How much Data is getting Indexed, what is the CPU/Memory on Splunk server, License volume being utilized, any spike in specific sourcetype ingesting more data etc. Even though dashboard might show situations like KPI above 80%, unless alerted/documented with proper alert action they might get missed or unnoticed.

Having said these,
1) You can schedule a dashboard for periodic pdf delivery in email (provided you have email exchange setup and your dashboard does not have interactive form elements).
2) You can also check out sendemail Splunk command to send out email if specific condition is met.

Re: Splunk Alerts & Dashboard Panels

martin_mueller — Tue, 11 Apr 2017 08:02:10 GMT

For the search itself

index=_internal host=*hfw* source=*metrics.log group=queue name=tcpout*

you'd wrap that in an eventtype with tags for re-use across multiple searches. Your eval calls would be best stored in a calculated field for that sourcetype, then you won't have to add them to every search. All that remains (where or timechart) is specific to the alert or dashboard, no need to change that.

Re: Splunk Alerts & Dashboard Panels

craigwilkinson — Tue, 11 Apr 2017 23:45:02 GMT

Ahh ok. So just to clarify, it is common to the have 2 separate scheduled searches running:
1st - to display the current queue size metric on a dashboard panel (and update as specified - using timechart)
2nd - to send an alert when the capacity exceeds 80%. (using "where")

It kind of seems inefficient to me - having two searches for essentially the same output.

The alerts are to be used for OOB hours, and when staff are away from their desks - where as the dashboard is to display pretty metrics during the office for management etc.

Re: Splunk Alerts & Dashboard Panels

martin_mueller — Wed, 12 Apr 2017 05:42:39 GMT

If you're truly worried about running over the raw data twice, you can define an accelerated data model on your data and power your dashboards and alerts off that.

In reality, you won't usually see big benefits just from two consuming searches though... especially considering your metrics will have under 3k events per host per day, and you're only looking at ten minutes - twenty events per host.