Splunk Search

Scheduling searches and delayed events

hcorleyss
New Member

Hi, is there a best practice to achieve the following?

I am looking to search for events and then to output them to a csv on an hourly schedule but want to ensure that any events that are delayed are also caught and sent in the next schedule.

An example would be:

All events from source X arriving between 8:00 and 9:00 are exported to csv by scheduled search.

An event from source X that has a time stamp in the event of 8:30 misses this search windows and is indexed at 9:30.

How can I ensure that the 8:30 event is included in the scheduled search between 9:00 and 10:00 but that any previous events that were sent between 8:00 and 9:00 are not sent again.

Is it best to use the index time rather than extract the time from the event during indexing?

Is there a way to add a tag or field to each event so that I know when it has previously been exported during the scheduled search and can use this to exclude from subsequent searches?

Tags (1)
0 Karma
1 Solution

somesoni2
SplunkTrust
SplunkTrust

If you've your logs coming not at real-time (you can get yesterday's data today), then it would be better to use the index time in your searches. Find out what is the maximum latency you can have (say it's 30 days), set your search time range to that (OR use All times) and include index time filter.

index=foo sourcetype=blah earliest=-30 _index_earliest=-1h@h _index_latest=@h | rest of the search

View solution in original post

0 Karma

somesoni2
SplunkTrust
SplunkTrust

If you've your logs coming not at real-time (you can get yesterday's data today), then it would be better to use the index time in your searches. Find out what is the maximum latency you can have (say it's 30 days), set your search time range to that (OR use All times) and include index time filter.

index=foo sourcetype=blah earliest=-30 _index_earliest=-1h@h _index_latest=@h | rest of the search
0 Karma

sundareshr
Legend

What if you started your scheduled job at 15 mins after the hour and have your search return data from the previous hour (earliest=-1h@h latest=@h).

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...