Alerting

How to re-run a scheduled correlation search after planned downtime?

sheamus69
Communicator

Where there is a planned scheduled outage of a network device, which will effectively kill many of the feeds due to come into splunk, what would be the best approach to re-running correlation searches on the missing feeds once connection has been restored?

0 Karma
1 Solution

kamal_jagga
Contributor

You can use the curl command to trigger the searches from back-end without changing its properties.
eg. curl -k -u username:Password https://localhost:9086/services/saved/searches/testkamal2/reschedule -X POST

This will immediately trigger your search. You can write a script to trigger the queries one by one every few mins.

View solution in original post

kamal_jagga
Contributor

You can use the curl command to trigger the searches from back-end without changing its properties.
eg. curl -k -u username:Password https://localhost:9086/services/saved/searches/testkamal2/reschedule -X POST

This will immediately trigger your search. You can write a script to trigger the queries one by one every few mins.

sheamus69
Communicator

How would this work for effectively looking at the missed data from the restored feed? Surely that command would only run the search now, not step back through the data? Or am I missing something obvious (admittedly, highly likely)?

0 Karma

FrankVl
Ultra Champion

Aren't correlation searches real time searches that run based on index time, rather than event time?

0 Karma

sheamus69
Communicator

Nope, they run at scheduled intervals, and if the event feed is delayed, things can be missed.

0 Karma

sheamus69
Communicator

Just did a little reading into this approach, and I think you are right, as shown in the example from the rest api page:

curl -k -u admin:pass https://localhost:8089/services/saved/searches/Purchased%20products%2C%20last%2024%20hours/reschedul... -d schedule_time=2012-08-15T14:11:01Z

0 Karma
Get Updates on the Splunk Community!

Splunk is Nurturing Tomorrow’s Cybersecurity Leaders Today

Meet Carol Wright. She leads the Splunk Academic Alliance program at Splunk. The Splunk Academic Alliance ...

Part 2: A Guide to Maximizing Splunk IT Service Intelligence

Welcome to the second segment of our guide. In Part 1, we covered the essentials of getting started with ITSI ...

Part 1: A Guide to Maximizing Splunk IT Service Intelligence

As modern IT environments continue to grow in complexity and speed, the ability to efficiently manage and ...