Hi! Perhaps I'm a bit late to the party, but are you able to send alerts to Slack from alerts running in the Monitoring Console app?
I have a Slack integration which seemingly works just fine, but for some reason, alerts created inside the Monitoring Console app wont trigger the Slack webhook. I'm wondering if there are some special prerequisites for this specific app.
Just had to do this due to some crashing issues we're seeing post 5.0.2 upgrade. This seems to work for us:
index="_internal" source="*splunkd.log" host="your.search.head" "Unable to distribute to peer named" | rex field=_raw "Unable to distribute to peer named (?<indexer>.*):\d+ at " | dedup indexer
I found a way to alert:
index="_internal" source="*splunkd.log" Connect to X.X.X.X:9997 failed | table _time | eval Status= "Down" | eval Indexer = "SplunkIndexerName"
Here X.X.X.X is the ip of the splunk indexer.
It shows the ip in the search but am not able to extract it into my query. So am hardcoding the server name to show it in the results. Is there a way to convert the ip to server name?