Splunk Search

Scheduling searches and delayed events

hcorleyss
New Member

Hi, is there a best practice to achieve the following?

I am looking to search for events and then to output them to a csv on an hourly schedule but want to ensure that any events that are delayed are also caught and sent in the next schedule.

An example would be:

All events from source X arriving between 8:00 and 9:00 are exported to csv by scheduled search.

An event from source X that has a time stamp in the event of 8:30 misses this search windows and is indexed at 9:30.

How can I ensure that the 8:30 event is included in the scheduled search between 9:00 and 10:00 but that any previous events that were sent between 8:00 and 9:00 are not sent again.

Is it best to use the index time rather than extract the time from the event during indexing?

Is there a way to add a tag or field to each event so that I know when it has previously been exported during the scheduled search and can use this to exclude from subsequent searches?

Tags (1)
0 Karma
1 Solution

somesoni2
Revered Legend

If you've your logs coming not at real-time (you can get yesterday's data today), then it would be better to use the index time in your searches. Find out what is the maximum latency you can have (say it's 30 days), set your search time range to that (OR use All times) and include index time filter.

index=foo sourcetype=blah earliest=-30 _index_earliest=-1h@h _index_latest=@h | rest of the search

View solution in original post

0 Karma

somesoni2
Revered Legend

If you've your logs coming not at real-time (you can get yesterday's data today), then it would be better to use the index time in your searches. Find out what is the maximum latency you can have (say it's 30 days), set your search time range to that (OR use All times) and include index time filter.

index=foo sourcetype=blah earliest=-30 _index_earliest=-1h@h _index_latest=@h | rest of the search
0 Karma

sundareshr
Legend

What if you started your scheduled job at 15 mins after the hour and have your search return data from the previous hour (earliest=-1h@h latest=@h).

0 Karma
Get Updates on the Splunk Community!

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...

What's New in Splunk Cloud Platform 9.0.2208?!

Howdy!  We are happy to share the newest updates in Splunk Cloud Platform 9.0.2208! Analysts can benefit ...