Alerting

Alert action script returned error codes on webhook alert.

n0vsec
Explorer

I have set up an alert using webhooks and they have not been firing.

I have set the notification to also show up in triggered alerts to make sure that the alert was in fact firing. When looking through the logs via index=_* webhook action=webhook I found some errors, which I cannot figure out how to remediate:

event_message: action=webhook - Alert action script returned error code=3
event_message: action=webhook - Alert action script completed in duration=64 ms with exit code=3
event_message: action=webhook STDERR - Unexpected error: POST data should be bytes, an iterable of bytes, or a file object. It cannot be of type str.

I realize that the last one is a Python error, which I found some information on here: stackoverflow - Python 3 urllib produces TypeErr...

I guess what I am wondering is if this is something that I might be doing wrong? Or is something broken on the cloud platform?

Labels (2)
1 Solution

n0vsec
Explorer

So apparently at some point in time there was an update to our Splunk Cloud instance that broke webhooks, the only way to fix this was to put a ticket in and revert to Python 2.7 until the following update.  Not the best solution I would say, but everything is working for now.

View solution in original post

0 Karma

sutanunandigram
Explorer

There is a workaround that worked for me without reverting back python version.

 

Go to the following location < Splunk\etc\apps\alert_webhook\bin> and you will find the webhook python script.

Add the line I have highlighted below. Then restart Splunk.

sutanunandigram_0-1626854719136.png

 

Tags (1)

kilianw
Engager

Did you manage to find a resolution?

n0vsec
Explorer

So apparently at some point in time there was an update to our Splunk Cloud instance that broke webhooks, the only way to fix this was to put a ticket in and revert to Python 2.7 until the following update.  Not the best solution I would say, but everything is working for now.

0 Karma

kilianw
Engager

Downgrading python version wasn't an option, so we are forking the webhook code and applying a fix to /etc/apps/alert_webhook/bin/webhook.py

Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...