topic Re: Alert action script returned error codes on webhook alert. in Alerting

Alert action script returned error codes on webhook alert.

n0vsec — Tue, 09 Jun 2020 17:16:17 GMT

I have set up an alert using webhooks and they have not been firing.

I have set the notification to also show up in triggered alerts to make sure that the alert was in fact firing. When looking through the logs via index=_* webhook action=webhook I found some errors, which I cannot figure out how to remediate:

event_message: action=webhook - Alert action script returned error code=3
event_message: action=webhook - Alert action script completed in duration=64 ms with exit code=3
event_message: action=webhook STDERR - Unexpected error: POST data should be bytes, an iterable of bytes, or a file object. It cannot be of type str.

I realize that the last one is a Python error, which I found some information on here: stackoverflow - Python 3 urllib produces TypeErr...

I guess what I am wondering is if this is something that I might be doing wrong? Or is something broken on the cloud platform?

Re: Alert action script returned error codes on webhook alert.

kilianw — Thu, 27 Aug 2020 13:25:10 GMT

Did you manage to find a resolution?

Re: Alert action script returned error codes on webhook alert.

n0vsec — Mon, 31 Aug 2020 15:39:29 GMT

So apparently at some point in time there was an update to our Splunk Cloud instance that broke webhooks, the only way to fix this was to put a ticket in and revert to Python 2.7 until the following update. Not the best solution I would say, but everything is working for now.

Re: Alert action script returned error codes on webhook alert.

kilianw — Fri, 18 Sep 2020 10:12:08 GMT

Downgrading python version wasn't an option, so we are forking the webhook code and applying a fix to /etc/apps/alert_webhook/bin/webhook.py

Re: Alert action script returned error codes on webhook alert.

sutanunandigram — Wed, 21 Jul 2021 08:06:02 GMT

There is a workaround that worked for me without reverting back python version.

Go to the following location < Splunk\etc\apps\alert_webhook\bin> and you will find the webhook python script.

Add the line I have highlighted below. Then restart Splunk.