To Monitor Splunk, we are leveraging Splunk Management Console, Deployment Server and integrating these Alerts with ServiceNow for auto-ticketing. i want to know details on different error conditions we can setup alerts so we detect some issues beforehand.
Also, is there any automation approach tried like if one Splunk server is running low on space, scripts can be triggered to clean up space on that server?
If you edit the alerts built into the Monitoring Console, it is possible to trigger a script to run in addition to posting the alert and sending an email. Since the script runs on the OS (either a bash script or a bat script or a Powershell script), you could have the script take a variety of actions.
The old scripted alert action in Splunk is still available, although it has been deprecated. Documentation here.
However, the new custom alert action is even more powerful. Documentation here. A number of custom alert actions are downloadable from Splunkbase as of today:
Custom Alert Action Webhook Plus
Slackbot Custom Alert
Slack Notification Alert
Thanks Iguinn. With ServiceNow Splunk App, Monitoring Console alerts can be integrated with ServiceNow for auto-ticketing. right ?