Splunk Search

How to perform incremental search

hmlathigara
Observer


I have some events data in which I have fields like Eventid, EventTime, EventRunId, AccountID etc. As per my use case I need to export the search result data to AWS S3 and for that I am using "Event Push by Deductiv" app.

In this app I have written below query and that is converting all search results to .csv file and send to S3 bucket.

 

 

 

 

source="events.csv" host="pool150.info.com" sourcetype="csv" | s3ep credential=default_password outputfile="eventcsv.csv" outputformat=csv compression=false fields="accountId,eventId,eventrunId,EventTime"

 

 

 

 

My purpose is to write a search query through which I can export search results in incremental way. We can create an alert running every hour.

For ex.

Let's say I have searched data just now (15:00 GMT+5:30 ) and I got a record with latest EventTime 14:53 GMT+5:30.
If I search the data again after 1 hour ( 16:00 GMT +5:30 ) then It should bring only records with EventTime > 14:53 PM GMT+5:30 and EventTime<=16:00 GMT +5:30 .

We might need a variable to store the latest event time somewhere and then compare to the upcoming records.Also, in case the splunk server is restarted or faced breakdown then after the server is up, it should pick up/persist the latest EventTime to carry on the incremental search.
My approach may be different and you can have some better solution as well.

Please help me.

 

Labels (2)
0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

You could try storing the latest time from the search in a key store/csv file and use that as the earliest time for the next time the search is executed.

0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...