Alerting

How to re-run a scheduled correlation search after planned downtime?

sheamus69
Communicator

Where there is a planned scheduled outage of a network device, which will effectively kill many of the feeds due to come into splunk, what would be the best approach to re-running correlation searches on the missing feeds once connection has been restored?

0 Karma
1 Solution

kamal_jagga
Contributor

You can use the curl command to trigger the searches from back-end without changing its properties.
eg. curl -k -u username:Password https://localhost:9086/services/saved/searches/testkamal2/reschedule -X POST

This will immediately trigger your search. You can write a script to trigger the queries one by one every few mins.

View solution in original post

kamal_jagga
Contributor

You can use the curl command to trigger the searches from back-end without changing its properties.
eg. curl -k -u username:Password https://localhost:9086/services/saved/searches/testkamal2/reschedule -X POST

This will immediately trigger your search. You can write a script to trigger the queries one by one every few mins.

sheamus69
Communicator

How would this work for effectively looking at the missed data from the restored feed? Surely that command would only run the search now, not step back through the data? Or am I missing something obvious (admittedly, highly likely)?

0 Karma

FrankVl
Ultra Champion

Aren't correlation searches real time searches that run based on index time, rather than event time?

0 Karma

sheamus69
Communicator

Nope, they run at scheduled intervals, and if the event feed is delayed, things can be missed.

0 Karma

sheamus69
Communicator

Just did a little reading into this approach, and I think you are right, as shown in the example from the rest api page:

curl -k -u admin:pass https://localhost:8089/services/saved/searches/Purchased%20products%2C%20last%2024%20hours/reschedul... -d schedule_time=2012-08-15T14:11:01Z

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...