Automated monitoring of splunk dashboards and alerts

[sameerdeepu2000]

Hi All, 

I am from an application production support team and we use splunk as our monitoring tool along with other tools. We use splunk primarily to get an understanding of the user actions via logs. 

We built some traditional dashboards and alerts to enhance our monitoring. We do our application health checks which include manually looking at splunk dashboards to see any spike in errors. 

I would like to automate this step where we check the dashboards and report them if there are any queries on dashboards that are trending red. Preferably post a RGB status on teams chat / email

Any leads on how to build this solution is much appreciated.

[inventsekar]

Hi @sameerdeepu2000 ... you can create Splunk Alerts easily.. 

let me give you an example...

1) simply run a splunk search query (index=User_Custom_Index username=testUser  | stats count by username)

2) you can save that search query as as alert ..... ( just above the time-picker... "Save As"..choose "Alert" in the drop-down)

3) Splunk Alert gives you options to send email alerts.. for example.. if the count by a user is above 10, you can send email alert to your team DL. 

pls find doc link... https://docs.splunk.com/Documentation/Splunk/latest/Alert/Aboutalerts

[sameerdeepu2000]

Thanks @inventsekar . Apologies if my question was unclear but we do have alerts and dashboards configured. What we want now is automated health check in a simple RGB status which tells status of dashboards and alerts. 

[PickleRick]

Ok, but what would be the purpose of such check? What should it do?

[sameerdeepu2000]

The purpose of these dashboards are health checks. We do manually check these dashboards to see if the errors are within the thresholds. If they breach, we check if there is any actual issue going on. 
Though we have alerts configured, we do these checks manually 6 times a day, to ensure stability. 

we would like to move away manual checks and see for any automation options 

[inventsekar]

Generally speaking..

1) Alerts are created for monitoring a threshold and get email notification (or other tasks like incident creation, etc)

2) Reports are created for daily/weekly/monthly reports generation(generally on a large dataset) and email the reports. 

3) Dashboards are created for viewing/checking/showcasing the current status of a search query/system. 

So generally you will not required email notification from dashboard. hope you got it. thanks. 

As you are a new member, let me update you that, karma points / upvotes are appreciated by everybody. thanks.