How to Export large amount (GBs) of data from Splunk?

[rgarcia]

Hello,

I'm trying to export, dump, or download large quantity of data from splunk. So far I tried dump command and the splunk cli search command to do this

-When I ran the search in the UI followed by the dump command and once the search finished I was unable to locate the file. Place I look for was /opt/splunk/var/run/splunk/dispatch, but I may be looking in the wrong system...is it my indexer or searchhead where this file is located?

-using the cli search command created some memory issues or login failures

Other options?

Note:I am the Splunk Admin, 6 indexer, 6 searchheads

[thambisetty]

@rgarcia 

file will be created on the node where you run the search. I believe you run the search on search head so you will see file in search head only.

I found GUI option is best when dumping raw events in GBs.

Below is the query I use to dump logs from Splunk to file:

index=foo| eval _dstpath=strftime(_time, "%Y%m%d") | dump basefilename=zscaler

_dstpath specifies the directory format under dump directory example below : 20180124

basefilename is the filename starts with under $SPLUNK_HOME/var/run/splunk/dispatch/<sidfrominspectjob>/dump/20180124/zscaler****

use the above search to create a dump for the time range you have chosen : if you choose two days lets say 24 and 25 of Jan, directories will be created like below:

/opt/splunk/var/run/splunk/dispatch/<sidfrominspectjob>/dump/20180124
/opt/splunk/var/run/splunk/dispatch/<sidfrominspectjob>/dump/20180125

make sure that you have enough space on the search head to run search for long time ranges. if you don't have more space on search head then run a search by choosing one day from time picker.

[rgarcia]

I tried the GUI and is just not feasible as the export is in txt format (limitations).

Is there a way to just copy (create a backup) or tar existing data locally without having to use queries or cli tools?

[rgarcia]

Hi  thambisetty

I tried the dump query and I can see the SID directory, but no dump or file is found after the search is completed (finished with 9 events).

This is the query I'm using index=something sourcetype=something | eval _dstpath=strftime(_time, "%Y%m%d") | dump basefilename=something_something

I will try to export using GUI and see how that works out for me.