Alerting

Alert action script returned error codes on webhook alert.

n0vsec
Explorer

I have set up an alert using webhooks and they have not been firing.

I have set the notification to also show up in triggered alerts to make sure that the alert was in fact firing. When looking through the logs via index=_* webhook action=webhook I found some errors, which I cannot figure out how to remediate:

event_message: action=webhook - Alert action script returned error code=3
event_message: action=webhook - Alert action script completed in duration=64 ms with exit code=3
event_message: action=webhook STDERR - Unexpected error: POST data should be bytes, an iterable of bytes, or a file object. It cannot be of type str.

I realize that the last one is a Python error, which I found some information on here: stackoverflow - Python 3 urllib produces TypeErr...

I guess what I am wondering is if this is something that I might be doing wrong? Or is something broken on the cloud platform?

Labels (2)
1 Solution

n0vsec
Explorer

So apparently at some point in time there was an update to our Splunk Cloud instance that broke webhooks, the only way to fix this was to put a ticket in and revert to Python 2.7 until the following update.  Not the best solution I would say, but everything is working for now.

View solution in original post

0 Karma

sutanunandigram
Explorer

There is a workaround that worked for me without reverting back python version.

 

Go to the following location < Splunk\etc\apps\alert_webhook\bin> and you will find the webhook python script.

Add the line I have highlighted below. Then restart Splunk.

sutanunandigram_0-1626854719136.png

 

Tags (1)

kilianw
Engager

Did you manage to find a resolution?

n0vsec
Explorer

So apparently at some point in time there was an update to our Splunk Cloud instance that broke webhooks, the only way to fix this was to put a ticket in and revert to Python 2.7 until the following update.  Not the best solution I would say, but everything is working for now.

0 Karma

kilianw
Engager

Downgrading python version wasn't an option, so we are forking the webhook code and applying a fix to /etc/apps/alert_webhook/bin/webhook.py

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...