https://community.splunk.com/t5/Getting-Data-In/How-to-ingest-large-volume-of-data-to-Splunk-via-REST-API-call/m-p/645694#M109852 <P>Hi Team! I need to make a REST API GET call to ingest a fairly large amount of data to splunk and unfortunately, this REST API doesn't have a pagination logic. So, it literally dumps an entire set of some 50,000 records every time. So may I know which can be the best way to ingest only new data into Splunk? Is there any best practices to ingest only delta records (which are new) to the system? Thanks.&nbsp; &nbsp;</P> Mon, 05 Jun 2023 14:18:39 GMT arjitg 2023-06-05T14:18:39Z https://community.splunk.com/t5/Getting-Data-In/How-to-ingest-large-volume-of-data-to-Splunk-via-REST-API-call/m-p/645694#M109852 <P>Hi Team! I need to make a REST API GET call to ingest a fairly large amount of data to splunk and unfortunately, this REST API doesn't have a pagination logic. So, it literally dumps an entire set of some 50,000 records every time. So may I know which can be the best way to ingest only new data into Splunk? Is there any best practices to ingest only delta records (which are new) to the system? Thanks.&nbsp; &nbsp;</P> Mon, 05 Jun 2023 14:18:39 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-ingest-large-volume-of-data-to-Splunk-via-REST-API-call/m-p/645694#M109852 arjitg 2023-06-05T14:18:39Z https://community.splunk.com/t5/Getting-Data-In/How-to-ingest-large-volume-of-data-to-Splunk-via-REST-API-call/m-p/645731#M109859 <P>Hi</P><P>Are you writing a modular inputs? If so, you should add a checkpoint method for bookkeeping what you have already ingesting and what is the next one.</P><UL><LI><A href="https://dev.splunk.com/enterprise/docs/developapps/manageknowledge/custominputs/modinputsscript/" target="_blank">https://dev.splunk.com/enterprise/docs/developapps/manageknowledge/custominputs/modinputsscript/</A></LI><LI><A href="https://dev.splunk.com/enterprise/docs/developapps/manageknowledge/custominputs/modinputsscript#Set-up-checkpointing" target="_blank">https://dev.splunk.com/enterprise/docs/developapps/manageknowledge/custominputs/modinputsscript#Set-up-checkpointing</A></LI></UL><P>If you are asking something else, then you must clarify your question.</P><P>r. Ismo</P> Mon, 05 Jun 2023 09:20:47 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-ingest-large-volume-of-data-to-Splunk-via-REST-API-call/m-p/645731#M109859 isoutamo 2023-06-05T09:20:47Z https://community.splunk.com/t5/Getting-Data-In/How-to-ingest-large-volume-of-data-to-Splunk-via-REST-API-call/m-p/645821#M109881 <P>Thanks&nbsp;<a href="https://community.splunk.com/t5/user/viewprofilepage/user-id/214410">@isoutamo</a>! Yeah So this is a scripted input but certainly, I can implement the checkpoint logic to this solution.&nbsp;</P> Mon, 05 Jun 2023 22:00:57 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-ingest-large-volume-of-data-to-Splunk-via-REST-API-call/m-p/645821#M109881 arjitg 2023-06-05T22:00:57Z