How to generate alerts programmatically?

[deshpandevikasv]

Hi,

I want to achieve this,
Whenever we search something in Splunk, I want to return the search url along with the splunk results.
Manually, I created an alert in Splunk which triggers every hour and returns the View Results link.
How can I do it programmatically? I don't want to create alerts manually like this.
Please help.

TIA.

[jhupka_splunk]

If you are wanting to generate your own link directly to the results you'll need to find out the Search ID (SID) of the query you just ran. The easiest way to do this is via the addinfo command. For example:

index=awesome_data earliest=-60m | ...  | addinfo

This will add a few fields to all of your event results, but the one you care about is info_sid. You can use this field to build your own reference to the results of the search. The search page within Splunk can take a form field named sid with the value of the Search ID like the following:

https://splunk.deshpandevikasv.com/en-US/app/search/search?sid=1501109999.63975

So if you wanted to do this within the search you could do something like this:

index=awesome_data earliest=-60m | ...  | addinfo | eval results_url="https://splunk.deshpandevikasv.com/en-US/app/search/search?sid=" . info_sid

When you go to the link with a provided sid then Splunk will find the results from that prior query from the dispatch directory.

A couple notes:

1. Keep in mind the default retention of search results. You don't want Splunk to delete the results after 10m if you want to access the data via URL for a couple days. If you're doing this via a Scheduled Search then make sure you set the TTL of the results appropriately. This will also impact disk space if you are saving lots of results for a long time.
2. You might not want to go directly to the Search page - but obtaining that sid using the addinfo command is the key take-away. That sid value can be used other places once you have it (e.g. using the loadjob command)

[deshpandevikasv]

Thank you @jhupka [Splunk]. Will try this as well.

[deshpandevikasv]

@jhupka, I am trying to replicate things which happen when we create an alert manually in Splunk and schedule it to run. When we run it and say the action as send email, we get View Results link.
For now it is displaying me the results in my console, I want to replace it with a Splunk link clicking on which the user is taken to splunk web ui.
Embedding the search id in url won't work for me since I am keeping this splunk alert real time and have saved the search only once.
I appreciate your time, Thanks!

[jhupka_splunk]

There's a few options available. With the built-in Email Alert Action you can control some of the formatting, add your own links (albeit hard to make dynamic), and change what is in the body with the default options.

On the other hand writing a Custom Alert Action in Splunk is fairly straight forward and plugs into an existing interface to make it easily configurable via the Splunk Web GUI:

https://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/ModAlertsLog

Also, there are many Custom Alert Action apps out on splunkbase that could fulfill your needs:

https://splunkbase.splunk.com/apps/#/search/alert%20action/

For example, if you do need more of an alert/incident workflow, there's an app for that:

https://splunkbase.splunk.com/apps/#/search/alert%20action/

Finally, from my experience I was never a fan of sending a basic email alert that just dumped people into Search with the raw results. I always preferred to use the alert as the notification of the problem, then the email would link them to a relevant dashboard that searched not only the data that might have triggered the alert, but also correlating data to immediately help with root-cause analysis. For example, if an order management system had orders failing, the dashboard would not only display what has failed and easily let them select criteria to group by server/customer/etc to see who is being affected with failures, but would also display *nix data of CPU, memory, and disk usage for the servers processing orders so common failure-modes are already being displayed to the person handling the alert to begin investigation.

[inventsekar]

Manually, I created an alert in Splunk which triggers every hour and returns the View Results link ///

may i know how you get this "View Results" link?
this is on Linux or windows?

[deshpandevikasv]

@inventsekar, I get the View Results link in the email which is the action set to work when the alert is triggered every hour. And, I am using Mac which shouldn't matter since this is just an alert triggered in splunk web ui and the action is to send an email.

[niketn]

@deshpandevikasv, can you please describe as to what you want to achieve/mean by "programatically"?

Sending email is one of the alert actions.

You can also send emails from
1) Dashboards: http://docs.splunk.com/Documentation/SplunkCloud/latest/Report/GeneratePDFsofyourreportsanddashboard...
2) Scheduled Reports : https://docs.splunk.com/Documentation/Splunk/latest/Report/Schedulereports#Define_a_Send_Email_actio...
3) sendemail SPL command: http://docs.splunk.com/Documentation/Splunk/latest/Alert/Emailnotification

[deshpandevikasv]

Thank you for the comment @niketnilay!
By programmatically I meant to create an alert through a java program depending on the input I get from another service.

[niketn]

@deshpandevikasv, Seems like you want to create the send email from outside of Splunk without having to create an Alert in Splunk.

You might have to go for either:
(1) Calling Splunk REST API to Authenticate and connect to Splunk and execute a SPLUNK search with sendemail command to trigger the email (http://docs.splunk.com/Documentation/Splunk/latest/RESTTUT/RESTsearches) or
(2) Use Splunk SDK for Java to perform similar activities (http://dev.splunk.com/java).

[deshpandevikasv]

Thank you @niketnilay. I was able to create an alert and if i run the same code again it gives me "Saved Search already exists error" which is expected. But I am not able to see that alert being created in Splunk web UI. Have any idea why?

TIA.

[umeshighe]

@deshpandevikasv can you share the code to create alert ?