Splunk Search

Scheduling searches and delayed events

hcorleyss
New Member

Hi, is there a best practice to achieve the following?

I am looking to search for events and then to output them to a csv on an hourly schedule but want to ensure that any events that are delayed are also caught and sent in the next schedule.

An example would be:

All events from source X arriving between 8:00 and 9:00 are exported to csv by scheduled search.

An event from source X that has a time stamp in the event of 8:30 misses this search windows and is indexed at 9:30.

How can I ensure that the 8:30 event is included in the scheduled search between 9:00 and 10:00 but that any previous events that were sent between 8:00 and 9:00 are not sent again.

Is it best to use the index time rather than extract the time from the event during indexing?

Is there a way to add a tag or field to each event so that I know when it has previously been exported during the scheduled search and can use this to exclude from subsequent searches?

Tags (1)
0 Karma
1 Solution

somesoni2
Revered Legend

If you've your logs coming not at real-time (you can get yesterday's data today), then it would be better to use the index time in your searches. Find out what is the maximum latency you can have (say it's 30 days), set your search time range to that (OR use All times) and include index time filter.

index=foo sourcetype=blah earliest=-30 _index_earliest=-1h@h _index_latest=@h | rest of the search

View solution in original post

0 Karma

somesoni2
Revered Legend

If you've your logs coming not at real-time (you can get yesterday's data today), then it would be better to use the index time in your searches. Find out what is the maximum latency you can have (say it's 30 days), set your search time range to that (OR use All times) and include index time filter.

index=foo sourcetype=blah earliest=-30 _index_earliest=-1h@h _index_latest=@h | rest of the search
0 Karma

sundareshr
Legend

What if you started your scheduled job at 15 mins after the hour and have your search return data from the previous hour (earliest=-1h@h latest=@h).

0 Karma
Get Updates on the Splunk Community!

Using Machine Learning for Hunting Security Threats

REGISTER NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more ...

Security Highlights | November 2022 Newsletter

 November 2022 2022 Gartner Magic Quadrant for SIEM: Splunk Named a Leader for the 9th Year in a RowSplunk is ...

Platform Highlights | November 2022 Newsletter

 November 2022 Skill Up on Splunk with our New Builder Tech Talk SeriesCan you build it? Yes you can! *play ...