Alerting

Alert action script returned error codes on webhook alert.

n0vsec
Explorer

I have set up an alert using webhooks and they have not been firing.

I have set the notification to also show up in triggered alerts to make sure that the alert was in fact firing. When looking through the logs via index=_* webhook action=webhook I found some errors, which I cannot figure out how to remediate:

event_message: action=webhook - Alert action script returned error code=3
event_message: action=webhook - Alert action script completed in duration=64 ms with exit code=3
event_message: action=webhook STDERR - Unexpected error: POST data should be bytes, an iterable of bytes, or a file object. It cannot be of type str.

I realize that the last one is a Python error, which I found some information on here: stackoverflow - Python 3 urllib produces TypeErr...

I guess what I am wondering is if this is something that I might be doing wrong? Or is something broken on the cloud platform?

Labels (2)
1 Solution

n0vsec
Explorer

So apparently at some point in time there was an update to our Splunk Cloud instance that broke webhooks, the only way to fix this was to put a ticket in and revert to Python 2.7 until the following update.  Not the best solution I would say, but everything is working for now.

View solution in original post

0 Karma

sutanunandigram
Explorer

There is a workaround that worked for me without reverting back python version.

 

Go to the following location < Splunk\etc\apps\alert_webhook\bin> and you will find the webhook python script.

Add the line I have highlighted below. Then restart Splunk.

sutanunandigram_0-1626854719136.png

 

Tags (1)

kilianw
Engager

Did you manage to find a resolution?

n0vsec
Explorer

So apparently at some point in time there was an update to our Splunk Cloud instance that broke webhooks, the only way to fix this was to put a ticket in and revert to Python 2.7 until the following update.  Not the best solution I would say, but everything is working for now.

0 Karma

kilianw
Engager

Downgrading python version wasn't an option, so we are forking the webhook code and applying a fix to /etc/apps/alert_webhook/bin/webhook.py

Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...