I have some events data in which I have fields like Eventid, EventTime, EventRunId, AccountID etc. As per my use case I need to export the search result data to AWS S3 and for that I am using "Event Push by Deductiv" app.
In this app I have written below query and that is converting all search results to .csv file and send to S3 bucket.
source="events.csv" host="pool150.info.com" sourcetype="csv" | s3ep credential=default_password outputfile="eventcsv.csv" outputformat=csv compression=false fields="accountId,eventId,eventrunId,EventTime"
My purpose is to write a search query through which I can export search results in incremental way. We can create an alert running every hour.
For ex.
Let's say I have searched data just now (15:00 GMT+5:30 ) and I got a record with latest EventTime 14:53 GMT+5:30.
If I search the data again after 1 hour ( 16:00 GMT +5:30 ) then It should bring only records with EventTime > 14:53 PM GMT+5:30 and EventTime<=16:00 GMT +5:30 .
We might need a variable to store the latest event time somewhere and then compare to the upcoming records.Also, in case the splunk server is restarted or faced breakdown then after the server is up, it should pick up/persist the latest EventTime to carry on the incremental search.
My approach may be different and you can have some better solution as well.
Please help me.
