Alerting

Slack Alert Customization

manish_singh_77
Builder

Hi Team,

How do we pass link result of splunk alerts in slack from splunk?

Labels (2)
Tags (1)
1 Solution

jkat54
SplunkTrust
SplunkTrust

jkat54
SplunkTrust
SplunkTrust
0 Karma

jkat54
SplunkTrust
SplunkTrust

Use this app and it's instructions:

https://splunkbase.splunk.com/app/2878/

manish_singh_77
Builder

Hi All,

I have configured Slack and email for the same alert, however we are not getting the same number of alerts in slack, alert has been scheduled to run every 2 minutes. Email alerts are coming properly but not in Slack. What could be an issue?

And there is also a delay in alerts that are coming on Slack. Quick help would be appreciated.

0 Karma

nickhills
Ultra Champion

Are you sure you are not hitting Slack's API rate limit?
Limits are applied to workspaces, so if you have several integrations (in yours and other channels) you could find that slack is limiting the number of messages delivered.

https://api.slack.com/docs/rate-limits

If my comment helps, please give it a thumbs up!
0 Karma

manish_singh_77
Builder

@nickhillscpl

How do we confirm whether rate limit is been applied to specific channel or not?

0 Karma

jkat54
SplunkTrust
SplunkTrust

By running the curl command....

0 Karma

manish_singh_77
Builder

How do we do that?

0 Karma

jkat54
SplunkTrust
SplunkTrust

I have shared the curl command above. Here is the doc I got it from:

https://api.slack.com/tutorials/slack-apps-hello-world

Please work with your server admin team if you don't know how to do this.

0 Karma

jkat54
SplunkTrust
SplunkTrust

jkat54 ♦ manish_singh_777 · Jan 27 at 12:54 PM
curl -X POST -H 'Content-type: application/json' --data '{"text":"Allow me to reintroduce myself!"}' YOUR_WEBHOOK_URL

0 Karma

manish_singh_77
Builder

@jkat54

Before I go ahead and accept this answer, need to know few things.

I am encountering the same issue again, alerts are coming to slack channel but there is a delay of 30 mins. For instance, if alert has been scheduled to run every 5 mins, In 30 mins I am getting only 2 alerts. Is it because of network issue?

0 Karma

jkat54
SplunkTrust
SplunkTrust

Check

  index=_internal sourcetype=scheduler "your saved search name"

See if there are any issues where the search is not returning expected results.

0 Karma

manish_singh_77
Builder

@jkat54

Do you think it is a connectivity issue?

0 Karma

jkat54
SplunkTrust
SplunkTrust

Splunk likes to skip searches when it's busy. That's my best guess.

0 Karma

manish_singh_77
Builder

@jkat54

I don't see error in the scheduler logs. Do you want me to search anything specific?

0 Karma

manish_singh_77
Builder

@jkat54

One more thing, I would like to highlight here is that, webhook created by me is now not visible under "Manage Webhook" page but still alerts are coming into splunk with the delay of 60 mins - 90 mins.

0 Karma

manish_singh_77
Builder

@jkat54

I have also observed that we are getting error code 255 only during the time interval when alerts were not received in Slack channel from Splunk.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Have you tried using curl from the server to post to slack?

Maybe there is packet loss or very slow connection to slack.

Or have you tried looking at the scheduler logs?

 index=_internal sourcetype=scheduler

Have you tried the job inspector to see if search.log has any errors when the alertS run?

0 Karma

manish_singh_77
Builder

@jkat54

We are getting alerts in slack but there is a delay in time, suppose I am scheculing an alert for 2 minutes for email and slack, in next 6 minutes I get 3 email alerts but only 2 slack alerts.

And in splunkd.log I see this error message "Alert action script returned error code=255". What could be an issue?

0 Karma

jkat54
SplunkTrust
SplunkTrust

Sounds like connectivity issues between your search head(s) and slack rest endpoints.

Have you tried using curl from the server to post to slack?

0 Karma

manish_singh_77
Builder

@jkat54

How do we do that?
Which command to run and from where?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...