i am trying to pull the data from splunk index using python and it triggers every 5 min. So i need to fetch the new data for the every run , nothing but an incremental data pull.
Splunk does not keep anything like a record number, to allow you to track the last record pulled and continue from there.
Since events can arrive well after the were generated, using the events time as the filter will cause you to miss events received late.
Start=$End_Previous_Run$+1
End=now()-5m
Then using those timeframes in your splunk search:
index=* criteria _index_earliest=$Start _index_latest=$End
Doing this should simulate a incremental pull of the data.
The format of the date sent can be in epoch time, which I recommend:
https://docs.splunk.com/Documentation/Splunk/7.2.6/SearchReference/SearchTimeModifiers
Splunk does not keep anything like a record number, to allow you to track the last record pulled and continue from there.
Since events can arrive well after the were generated, using the events time as the filter will cause you to miss events received late.
Start=$End_Previous_Run$+1
End=now()-5m
Then using those timeframes in your splunk search:
index=* criteria _index_earliest=$Start _index_latest=$End
Doing this should simulate a incremental pull of the data.
The format of the date sent can be in epoch time, which I recommend:
https://docs.splunk.com/Documentation/Splunk/7.2.6/SearchReference/SearchTimeModifiers
Thanks for the answer. In what format should i pass the date ?
I would use Epoch or Unix time:
https://en.wikipedia.org/wiki/Unix_time