Alerting

How to debug why Webhook alert is not triggered?

ramabu
Path Finder

Hi

I am using Splunk 6.3.1, a trial Splunk Enterprise.
I created a web-servelet in my app, and verified I can trigger the desired behavior with "curl" from the Splunk server command line, as well as from the google RESTApi application.

Then I ran a search, made sure it has results, and saved it as alert. (hourly at xx:45), but I don't see it happen.
I also created another, with same search but schedule at xx:15 - but no luck.

In Alerts => "Open in Search", I see the coorect search, with results, But I see no evidence of a triggered alert.

How can I debug why, or how far into the alert did the chain execute before it broke?

Thanks, rama

0 Karma
1 Solution

jplumsdaine22
Influencer

in Alerts , open the Alert by clicking the name of it.

First check that the alert is enabled.
Second check the trigger condition you have set
Third check the URL in your webhook action

If that all looks good I would add an additional action to the search, "Add to Triggered Alerts". You will then be able to see when the alert is triggered in the Alerts page. If you don't ever see the alert triggered then there is something wrong with the Trigger Condition for the alert.

If the alert is triggering ok, then examine the webhook url you have entered. If you can reach that from the splunk server then I would check your logs in your app to make sure that the requests are in the form you expect.

My guess would be here that either:
A) Your alert does not trigger
B) Your app is not parsing the JSON from Splunk correctly

View solution in original post

ramabu
Path Finder

I figured it out. I meant to post this yesterday, but "You are only allowed to submit 2 posts per day until you reach 40 points of reputation level." LOL.

jplumsdaine22 - you were extremely helpful.
When I added the "add to triggered events". I realize they we triggered.
Then I saw in my apache access logs I replied 401 to the posts. This would be because of the authorization token, of my servlet.
I put it into the URL (and chaged the service to accept that as well).

I am good to go.

Thanks for all the help!

0 Karma

jplumsdaine22
Influencer

No problem @ramabu - I sent you some extra points too.

Good luck with your Splunk!

jplumsdaine22
Influencer

in Alerts , open the Alert by clicking the name of it.

First check that the alert is enabled.
Second check the trigger condition you have set
Third check the URL in your webhook action

If that all looks good I would add an additional action to the search, "Add to Triggered Alerts". You will then be able to see when the alert is triggered in the Alerts page. If you don't ever see the alert triggered then there is something wrong with the Trigger Condition for the alert.

If the alert is triggering ok, then examine the webhook url you have entered. If you can reach that from the splunk server then I would check your logs in your app to make sure that the requests are in the form you expect.

My guess would be here that either:
A) Your alert does not trigger
B) Your app is not parsing the JSON from Splunk correctly

ramabu
Path Finder

Hi - and thank you all for the replies

So - I put up a test to figure this out better

I have a script that pushes specific data to splunk("pleaseAlertMe...") in a loop (sleep 5sec),
and an alert (webhook demo) that
* scheduled every 5min
* search for these events (index=fsctcenter ctupdate=notif pleaseAlertMe*)
* Adds to triggered events list
* does a webhook post once per result

And in the capture, I see no such posts.

Next

I looked at the results of a search:
index=_internal source=*scheduler.log savedsearch_name="webhook demo"

And I see multiple elements such as the following

01-27-2016 13:16:09.990 +0200 INFO SavedSplunker - savedsearch_id="nobody;fsctcenter;webhook demo", user="admin", app="fsctcenter", savedsearch_name="webhook demo", status=success, digest_mode=0, scheduled_time=1453893360, window_time=0, dispatch_time=1453893368, run_time=1.011, result_count=0, alert_actions="", sid="scheduler_adminfsctcenter_RMD52e17e279d4c1644c_at_1453893360_4063", suppressed=0, fired=0, skipped=0, action_time_ms=8, thread_id="AlertNotifierWorker-0", message=""

Note that it says 'result_count=0', and 'alert_actions=""', and 'fired=0'.
Not sure what they all mean, but there appear to be no-results;
however the count serach shows hundreds (index=fsctcenter ctupdate=notif pleaseAlertMe* | stats count by ctupdate)

Any new ideas?

0 Karma

renjith_nair
Legend

I think alerting is not enabled in trial licenses

See here :http://www.splunk.com/en_us/products/splunk-enterprise/free-vs-enterprise.html

---
What goes around comes around. If it helps, hit it with Karma 🙂
0 Karma

ramabu
Path Finder

I downvoted this post because clicked by mistake

0 Karma

jplumsdaine22
Influencer

All features should work in the Enterprise trial until its converted to free. You can test a distributed search cluster on a trial license for example

0 Karma

renjith_nair
Legend

Yeah, aware of that. It's a mere guess since there are no signs of alert

---
What goes around comes around. If it helps, hit it with Karma 🙂
0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...