Archive

jobs expire too soon when I export data via REST API

Communicator

Hello, I was trying and trying to export the data via REST API. I followed all the instructions from this thread:

https://www.splunk.com/blog/2013/09/15/exporting-large-results-sets-to-csv.html

But I see the jobs expire too soon when I export large data and I never get all the data I want because it sticks at 14% and 21%. I really don't know what to do. Is it a way to extend job expiration via curl or anything like that?

alt text

Tags (1)
0 Karma

Super Champion

This blog posts describes places where you can change the limits of the ttl of your searches

https://www.splunk.com/blog/2012/09/12/how-long-does-my-search-live-default-search-ttl.html

0 Karma

Communicator

Hi, I did everything in that post and I couldn't extend the lifetime. I just entered in the Indexer and extended the job lifetime manually in "Job settings". It is enough to me but If you know another way more automatically I would appreciate it.

0 Karma

Motivator

modify expiration field value from "after 24 hours" to some realistic value for your savedsearch/report.

0 Karma

Communicator

Hello @sbbadri. Could you be a bit more specific? I did not understand what you meant. I appreciated your help.

I retake this topic because I have more time to learn more about this.

Thank you everybody 😄

0 Karma

SplunkTrust
SplunkTrust

If you refer to the limits.conf documentation and the alert_actions.conf they both have TTL settings that you can change.

However if you want to export large amounts of data why not use the CLI?

curl -k -u admin:changeme https://localhost:8089/services/search/jobs/export -d search="search index=_internal earliest=-2s" -d output_mode=csv > ....(or similar)

I find the CLI interface much more efficient for large exports, the above will dump the data directly into a file...(which is probably what you are trying to do)

0 Karma

Communicator

Hi garethatiag, I exported the data via CLI but I always had the problem with jobs expiration because time is too short. My solved was editing the job settings manually and extend the lifetime, it was ok at the moment I needed but If I want something more automatically it's not the best choice.

I changed the TTL in those files and I did not get what I wanted, when I used the CLI the job time expiration was too short. I used to export 200GB of data.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!