Hi
I am using Splunk 6.3.1, a trial Splunk Enterprise.
I created a web-servelet in my app, and verified I can trigger the desired behavior with "curl" from the Splunk server command line, as well as from the google RESTApi application.
Then I ran a search, made sure it has results, and saved it as alert. (hourly at xx:45), but I don't see it happen.
I also created another, with same search but schedule at xx:15 - but no luck.
In Alerts => "Open in Search", I see the coorect search, with results, But I see no evidence of a triggered alert.
How can I debug why, or how far into the alert did the chain execute before it broke?
Thanks, rama
in Alerts , open the Alert by clicking the name of it.
First check that the alert is enabled.
Second check the trigger condition you have set
Third check the URL in your webhook action
If that all looks good I would add an additional action to the search, "Add to Triggered Alerts". You will then be able to see when the alert is triggered in the Alerts page. If you don't ever see the alert triggered then there is something wrong with the Trigger Condition for the alert.
If the alert is triggering ok, then examine the webhook url you have entered. If you can reach that from the splunk server then I would check your logs in your app to make sure that the requests are in the form you expect.
My guess would be here that either:
A) Your alert does not trigger
B) Your app is not parsing the JSON from Splunk correctly
I figured it out. I meant to post this yesterday, but "You are only allowed to submit 2 posts per day until you reach 40 points of reputation level." LOL.
jplumsdaine22 - you were extremely helpful.
When I added the "add to triggered events". I realize they we triggered.
Then I saw in my apache access logs I replied 401 to the posts. This would be because of the authorization token, of my servlet.
I put it into the URL (and chaged the service to accept that as well).
I am good to go.
Thanks for all the help!
No problem @ramabu - I sent you some extra points too.
Good luck with your Splunk!
in Alerts , open the Alert by clicking the name of it.
First check that the alert is enabled.
Second check the trigger condition you have set
Third check the URL in your webhook action
If that all looks good I would add an additional action to the search, "Add to Triggered Alerts". You will then be able to see when the alert is triggered in the Alerts page. If you don't ever see the alert triggered then there is something wrong with the Trigger Condition for the alert.
If the alert is triggering ok, then examine the webhook url you have entered. If you can reach that from the splunk server then I would check your logs in your app to make sure that the requests are in the form you expect.
My guess would be here that either:
A) Your alert does not trigger
B) Your app is not parsing the JSON from Splunk correctly
Hi - and thank you all for the replies
So - I put up a test to figure this out better
I have a script that pushes specific data to splunk("pleaseAlertMe...") in a loop (sleep 5sec),
and an alert (webhook demo) that
* scheduled every 5min
* search for these events (index=fsctcenter ctupdate=notif pleaseAlertMe*)
* Adds to triggered events list
* does a webhook post once per result
And in the capture, I see no such posts.
Next
I looked at the results of a search:
index=_internal source=*scheduler.log savedsearch_name="webhook demo"
And I see multiple elements such as the following
01-27-2016 13:16:09.990 +0200 INFO SavedSplunker - savedsearch_id="nobody;fsctcenter;webhook demo", user="admin", app="fsctcenter", savedsearch_name="webhook demo", status=success, digest_mode=0, scheduled_time=1453893360, window_time=0, dispatch_time=1453893368, run_time=1.011, result_count=0, alert_actions="", sid="scheduler_adminfsctcenter_RMD52e17e279d4c1644c_at_1453893360_4063", suppressed=0, fired=0, skipped=0, action_time_ms=8, thread_id="AlertNotifierWorker-0", message=""
Note that it says 'result_count=0', and 'alert_actions=""', and 'fired=0'.
Not sure what they all mean, but there appear to be no-results;
however the count serach shows hundreds (index=fsctcenter ctupdate=notif pleaseAlertMe* | stats count by ctupdate)
Any new ideas?
I think alerting is not enabled in trial licenses
See here :http://www.splunk.com/en_us/products/splunk-enterprise/free-vs-enterprise.html
I downvoted this post because clicked by mistake
All features should work in the Enterprise trial until its converted to free. You can test a distributed search cluster on a trial license for example
Yeah, aware of that. It's a mere guess since there are no signs of alert