topic Re: how can pass a date to search index to pull the incremental data in Splunk Search

how can pass a date to search index to pull the incremental data

nikilkatturi — Thu, 29 Aug 2019 14:47:20 GMT

i am trying to pull the data from splunk index using python and it triggers every 5 min. So i need to fetch the new data for the every run , nothing but an incremental data pull.

Re: how can pass a date to search index to pull the incremental data

solarboyz1 — Wed, 30 Sep 2020 01:57:37 GMT

Splunk does not keep anything like a record number, to allow you to track the last record pulled and continue from there.
Since events can arrive well after the were generated, using the events time as the filter will cause you to miss events received late.

Start=$End_Previous_Run$+1
End=now()-5m

Then using those timeframes in your splunk search:
index=* criteria _index_earliest=$Start _index_latest=$End

Doing this should simulate a incremental pull of the data.

The format of the date sent can be in epoch time, which I recommend:
https://docs.splunk.com/Documentation/Splunk/7.2.6/SearchReference/SearchTimeModifiers

Re: how can pass a date to search index to pull the incremental data

nikilkatturi — Tue, 10 Sep 2019 15:24:09 GMT

Thanks for the answer. In what format should i pass the date ?

Re: how can pass a date to search index to pull the incremental data

solarboyz1 — Tue, 10 Sep 2019 16:36:26 GMT

I would use Epoch or Unix time:
https://en.wikipedia.org/wiki/Unix_time