topic Re: Slack Alert Customization in Alerting

Slack Alert Customization

manish_singh_77 — Tue, 09 Jun 2020 01:11:55 GMT

Hi Team,

How do we pass link result of splunk alerts in slack from splunk?

Re: Slack Alert Customization

jkat54 — Fri, 17 Jan 2020 12:25:04 GMT

Use this app and it's instructions:

https://splunkbase.splunk.com/app/2878/

Re: Slack Alert Customization

manish_singh_77 — Fri, 24 Jan 2020 07:18:15 GMT

Hi All,

I have configured Slack and email for the same alert, however we are not getting the same number of alerts in slack, alert has been scheduled to run every 2 minutes. Email alerts are coming properly but not in Slack. What could be an issue?

And there is also a delay in alerts that are coming on Slack. Quick help would be appreciated.

Re: Slack Alert Customization

jkat54 — Fri, 24 Jan 2020 12:15:58 GMT

Have you tried using curl from the server to post to slack?

Maybe there is packet loss or very slow connection to slack.

Or have you tried looking at the scheduler logs?

 index=_internal sourcetype=scheduler

Have you tried the job inspector to see if search.log has any errors when the alertS run?

Re: Slack Alert Customization

manish_singh_77 — Mon, 27 Jan 2020 09:58:59 GMT

@jkat54

We are getting alerts in slack but there is a delay in time, suppose I am scheculing an alert for 2 minutes for email and slack, in next 6 minutes I get 3 email alerts but only 2 slack alerts.

And in splunkd.log I see this error message "Alert action script returned error code=255". What could be an issue?

Re: Slack Alert Customization

jkat54 — Mon, 27 Jan 2020 13:03:36 GMT

Sounds like connectivity issues between your search head(s) and slack rest endpoints.

Have you tried using curl from the server to post to slack?

Re: Slack Alert Customization

manish_singh_77 — Mon, 27 Jan 2020 18:15:44 GMT

@jkat54

How do we do that?
Which command to run and from where?

Re: Slack Alert Customization

jkat54 — Mon, 27 Jan 2020 20:54:10 GMT

curl -X POST -H 'Content-type: application/json' --data '{"text":"Allow me to reintroduce myself!"}' YOUR_WEBHOOK_URL

Re: Slack Alert Customization

manish_singh_77 — Wed, 29 Jan 2020 06:58:38 GMT

@jkat54

How can we run this command from search head url from the search query?

Re: Slack Alert Customization

jkat54 — Wed, 29 Jan 2020 07:53:16 GMT

It's a Linux command you run from a command line shell.

Re: Slack Alert Customization

manish_singh_77 — Wed, 30 Sep 2020 03:56:25 GMT

@jkat54

I tried different slack channel that is using the same webhook url for testing purpose and there also we observed the delay in receiving alerts.

PFB error message that we see in the splunkd.log

ERROR SearchScheduler - Error in 'sendalert' command: Alert script returned error code 255., search='sendalert slack_webhook_alert results_file="/opt/splunk/var/run/splunk/dispatch/scheduler_bXNpbmdoQGFybG8uY29t_c3BsdW5rX2FwcF9qZW5raW5z_RMD55de36a8bbf79c022_at_1580289000_23596_B2217163-89B0-464E-965F-D5D289E35131/results.csv.gz" results_link="********/app/splunk_app_jenkins/@go?sid=scheduler_bXNpbmdoQGFybG8uY29t_c3BsdW5rX2FwcF9qZW5raW5z_RMD55de36a8bbf79c022_at_1580289000_23596_B2217163-89B0-464E-965F-D5D289E35131"'

Re: Slack Alert Customization

nickhills — Wed, 29 Jan 2020 12:43:47 GMT

Are you sure you are not hitting Slack's API rate limit?
Limits are applied to workspaces, so if you have several integrations (in yours and other channels) you could find that slack is limiting the number of messages delivered.

https://api.slack.com/docs/rate-limits

Re: Slack Alert Customization

jkat54 — Wed, 29 Jan 2020 12:46:44 GMT

If you try the curl command... which is what slack says to do to test your webhook...:

It will display any errors such as a rate limit

Re: Slack Alert Customization

manish_singh_77 — Thu, 30 Jan 2020 08:01:14 GMT

@nickhillscpl

How do we confirm whether rate limit is been applied to specific channel or not?

Re: Slack Alert Customization

jkat54 — Thu, 30 Jan 2020 11:07:32 GMT

By running the curl command....

Re: Slack Alert Customization

manish_singh_77 — Thu, 13 Feb 2020 08:05:47 GMT

How do we do that?

Re: Slack Alert Customization

jkat54 — Thu, 13 Feb 2020 12:43:34 GMT

I have shared the curl command above. Here is the doc I got it from:

https://api.slack.com/tutorials/slack-apps-hello-world

Please work with your server admin team if you don't know how to do this.

Re: Slack Alert Customization

jkat54 — Wed, 30 Sep 2020 04:11:40 GMT

jkat54 ♦ manish_singh_777 · Jan 27 at 12:54 PM
curl -X POST -H 'Content-type: application/json' --data '{"text":"Allow me to reintroduce myself!"}' YOUR_WEBHOOK_URL

Re: Slack Alert Customization

manish_singh_77 — Tue, 19 May 2020 09:10:22 GMT

@jkat54

Before I go ahead and accept this answer, need to know few things.

I am encountering the same issue again, alerts are coming to slack channel but there is a delay of 30 mins. For instance, if alert has been scheduled to run every 5 mins, In 30 mins I am getting only 2 alerts. Is it because of network issue?

Re: Slack Alert Customization

jkat54 — Tue, 19 May 2020 12:47:16 GMT

Check

  index=_internal sourcetype=scheduler "your saved search name"

See if there are any issues where the search is not returning expected results.