Getting Data In

jobs expire too soon when I export data via REST API

jrballesteros05
Communicator

Hello, I was trying and trying to export the data via REST API. I followed all the instructions from this thread:

https://www.splunk.com/blog/2013/09/15/exporting-large-results-sets-to-csv.html

But I see the jobs expire too soon when I export large data and I never get all the data I want because it sticks at 14% and 21%. I really don't know what to do. Is it a way to extend job expiration via curl or anything like that?

alt text

Tags (1)
0 Karma

cmerriman
Super Champion

This blog posts describes places where you can change the limits of the ttl of your searches

https://www.splunk.com/blog/2012/09/12/how-long-does-my-search-live-default-search-ttl.html

0 Karma

jrballesteros05
Communicator

Hi, I did everything in that post and I couldn't extend the lifetime. I just entered in the Indexer and extended the job lifetime manually in "Job settings". It is enough to me but If you know another way more automatically I would appreciate it.

0 Karma

sbbadri
Motivator

modify expiration field value from "after 24 hours" to some realistic value for your savedsearch/report.

0 Karma

jrballesteros05
Communicator

Hello @sbbadri. Could you be a bit more specific? I did not understand what you meant. I appreciated your help.

I retake this topic because I have more time to learn more about this.

Thank you everybody 😄

0 Karma

gjanders
SplunkTrust
SplunkTrust

If you refer to the limits.conf documentation and the alert_actions.conf they both have TTL settings that you can change.

However if you want to export large amounts of data why not use the CLI?

curl -k -u admin:changeme https://localhost:8089/services/search/jobs/export -d search="search index=_internal earliest=-2s" -d output_mode=csv > ....(or similar)

I find the CLI interface much more efficient for large exports, the above will dump the data directly into a file...(which is probably what you are trying to do)

0 Karma

jrballesteros05
Communicator

Hi garethatiag, I exported the data via CLI but I always had the problem with jobs expiration because time is too short. My solved was editing the job settings manually and extend the lifetime, it was ok at the moment I needed but If I want something more automatically it's not the best choice.

I changed the TTL in those files and I did not get what I wanted, when I used the CLI the job time expiration was too short. I used to export 200GB of data.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...