Archive
Highlighted

Dynamic absolute time for alert search results

New Member

Background
We're currently running a Scheduled alert (pushing to Slack) with a simple Search query looking for "response=400", running every 5 minutes (cron'd)
- "Earliest" set to -5
- "Latest" set to Now
In the Alert body sent to Slack, we're returning the token URL (using $results_link$) to open up Splunk and dive in.

Problem
If we look at the Alert more than 5 minutes after the Alert is sent by clicking on the results link, there're no results displayed in Splunk (assuming no errors in the last 5 minutes). So as a workaround, we're currently adding "earliest=-24" to the Splunk query in the browser / entry field.

How can we preserve the date/time as to when the Search was run to have the results displayed in Splunk despite when the user clicked on the results link? Perhaps is there a way to pass the date/time from the result URL to be retrieved and/or some other mechanism configuration with date/time settings in the Search query? I looked at using time modifiers and didn't find anything suiting to this use case.

We have some issues with Real-Time alerting that we need to solve (internally), so ideally we can stay away from using that for the time being.

Thank you.

Tags (1)
0 Karma
Highlighted

Re: Dynamic absolute time for alert search results

Hi Jacob,

The link should not have earliest or latest in it, ideally it should refer to the existing job:
Something like:

https://splunk.instance/app/an_app/@go?sid=<job_id>;
0 Karma
Highlighted

Re: Dynamic absolute time for alert search results

Wait, is it a real-time search you are talking about?

0 Karma
Highlighted

Re: Dynamic absolute time for alert search results

New Member

@damien_chillet, no not a real time search. The alert is based on a Schedule to run every 5 mins.

0 Karma
Highlighted

Re: Dynamic absolute time for alert search results

Is the alert condition "Number of results > 0"?
If so the job results should be saved and clicking the link should display the results without having to run the job once more.

0 Karma
Highlighted

Re: Dynamic absolute time for alert search results

The saved results will expire shortly, since the alert is only looking back over 5 mins. I think the expected lifetime of a search like that is twice the interval over which it runs.

0 Karma
Highlighted

Re: Dynamic absolute time for alert search results

Yea it is like that for jobs. But in case of a triggered alert it should be kept longer, i think expiration time is 24h by default for a triggered alert.

0 Karma
Highlighted

Re: Dynamic absolute time for alert search results

New Member

Yes, the alert condition is set to "Number of results > 0". The behaviour I'm seeing is - if the link is clicked just after the alert is fired (within 1 min say), the results are displayed in Splunk. However, if the link is clicked >= 5 minutes after the alert fired (assuming there hasn't been an exception found since), then the results returned in Splunk are empty as the Relative search is 5 minutes ago.

0 Karma
Highlighted

Re: Dynamic absolute time for alert search results

Let's try with an example.
The search runs at 13:45 for events between 13:40 and 13:45.
It finds 1 error event so an alert link is sent to your slack channel.

Whatever the time is when you click the link it should load the job which ran at 13:45 for events between 13:40 and 13:45.

The time range would change only if you re-run the job manually.

0 Karma
Highlighted

Re: Dynamic absolute time for alert search results

New Member

Valid example, but I only see the results in Splunk if the link is opened just after 13:45 it seems, so I must be missing something if the time when you click the link shouldn't matter. Any other thoughts? Perhaps there's some back end configuration overriding the $resultslink$ parameters to re-run the job? Based on what @elliotproebstel was saying, the I need to pass 'infomintime' and 'infomax_time' into the Alert.

0 Karma