https://community.splunk.com/t5/Splunk-Search/the-search-results-disappear-how-to-keep-them-longer-python-SDK/m-p/77472#M19563 <P>Hi all,</P> <P>I'm writing a cron job (using the Python SDK) that does a search and exports the data to a CSV file (to analyze it in a different app). The result usually is a few hundreds thousands rows (~ a million or so), so I need to fetch it piecewise (by 50k). The problem is that sometimes the result suddenly disappears after fetching the first few segments.</P> <P>How to get rid of that? Is there an option to mark a result as 'do not remove' or something like that?</P> Mon, 18 Jun 2012 08:45:18 GMT tomasv 2012-06-18T08:45:18Z https://community.splunk.com/t5/Splunk-Search/the-search-results-disappear-how-to-keep-them-longer-python-SDK/m-p/77472#M19563 <P>Hi all,</P> <P>I'm writing a cron job (using the Python SDK) that does a search and exports the data to a CSV file (to analyze it in a different app). The result usually is a few hundreds thousands rows (~ a million or so), so I need to fetch it piecewise (by 50k). The problem is that sometimes the result suddenly disappears after fetching the first few segments.</P> <P>How to get rid of that? Is there an option to mark a result as 'do not remove' or something like that?</P> Mon, 18 Jun 2012 08:45:18 GMT https://community.splunk.com/t5/Splunk-Search/the-search-results-disappear-how-to-keep-them-longer-python-SDK/m-p/77472#M19563 tomasv 2012-06-18T08:45:18Z https://community.splunk.com/t5/Splunk-Search/the-search-results-disappear-how-to-keep-them-longer-python-SDK/m-p/77473#M19564 <P>Great question tomasv! The real answer has to do with the TTL of the search job. The default value is 600, and it unfortunately does not get reset when you get results from the job. You have a few options:</P> <OL> <LI>You can call <CODE>my_job.touch()</CODE> after every call to <CODE>my_job.results(...)</CODE> - that will reset the TTL.</LI> <LI>You can explicitly set a longer TTL with `my_job.set_ttl(6000)' (or some other number, of course).</LI> </OL> <P>The other thing to note is that if you truly have hundreds of thousands of rows you need to get out of a single search, you may be better suited with using the <CODE>search/jobs/export</CODE> endpoint. In the Python SDK, you can take a look at the export sample <A href="https://github.com/splunk/splunk-sdk-python/tree/master/examples/export" target="_blank">here</A>, which shows you how to do it. The benefit of the export endpoint is that it will simply stream the results to you as they are ready, so you don't have to keep paginating. You might find that this is a faster mechanism.</P> <P>Let me know if this makes sense, and if not, I can add some more details.</P> Mon, 28 Sep 2020 12:03:53 GMT https://community.splunk.com/t5/Splunk-Search/the-search-results-disappear-how-to-keep-them-longer-python-SDK/m-p/77473#M19564 ineeman 2020-09-28T12:03:53Z https://community.splunk.com/t5/Splunk-Search/the-search-results-disappear-how-to-keep-them-longer-python-SDK/m-p/77474#M19565 <P>Great answer ineeman! I haven't tried the export endpoint yet, but the touch() seems to be working just fine.</P> Thu, 09 Aug 2012 09:21:41 GMT https://community.splunk.com/t5/Splunk-Search/the-search-results-disappear-how-to-keep-them-longer-python-SDK/m-p/77474#M19565 tomasv 2012-08-09T09:21:41Z https://community.splunk.com/t5/Splunk-Search/the-search-results-disappear-how-to-keep-them-longer-python-SDK/m-p/77475#M19566 <P>thanks - let us know if you need anything else. Also, don't forget to select an answer.</P> Thu, 09 Aug 2012 15:21:40 GMT https://community.splunk.com/t5/Splunk-Search/the-search-results-disappear-how-to-keep-them-longer-python-SDK/m-p/77475#M19566 ineeman 2012-08-09T15:21:40Z