Alerting

Alert action script returned error codes on webhook alert.

n0vsec
Explorer

I have set up an alert using webhooks and they have not been firing.

I have set the notification to also show up in triggered alerts to make sure that the alert was in fact firing. When looking through the logs via index=_* webhook action=webhook I found some errors, which I cannot figure out how to remediate:

event_message: action=webhook - Alert action script returned error code=3
event_message: action=webhook - Alert action script completed in duration=64 ms with exit code=3
event_message: action=webhook STDERR - Unexpected error: POST data should be bytes, an iterable of bytes, or a file object. It cannot be of type str.

I realize that the last one is a Python error, which I found some information on here: stackoverflow - Python 3 urllib produces TypeErr...

I guess what I am wondering is if this is something that I might be doing wrong? Or is something broken on the cloud platform?

Labels (2)
1 Solution

n0vsec
Explorer

So apparently at some point in time there was an update to our Splunk Cloud instance that broke webhooks, the only way to fix this was to put a ticket in and revert to Python 2.7 until the following update.  Not the best solution I would say, but everything is working for now.

View solution in original post

0 Karma

sutanunandigram
Explorer

There is a workaround that worked for me without reverting back python version.

 

Go to the following location < Splunk\etc\apps\alert_webhook\bin> and you will find the webhook python script.

Add the line I have highlighted below. Then restart Splunk.

sutanunandigram_0-1626854719136.png

 

Tags (1)

kilianw
Engager

Did you manage to find a resolution?

n0vsec
Explorer

So apparently at some point in time there was an update to our Splunk Cloud instance that broke webhooks, the only way to fix this was to put a ticket in and revert to Python 2.7 until the following update.  Not the best solution I would say, but everything is working for now.

0 Karma

kilianw
Engager

Downgrading python version wasn't an option, so we are forking the webhook code and applying a fix to /etc/apps/alert_webhook/bin/webhook.py

Get Updates on the Splunk Community!

What’s new on Splunk Lantern in August

This month’s Splunk Lantern update gives you the low-down on all of the articles we’ve published over the past ...

Welcome to the Future of Data Search & Exploration

You have more data coming at you than ever before. Over the next five years, the total amount of digital data ...

This Week's Community Digest - Splunk Community Happenings [8.3.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...