Splunk Enterprise

How to Export large amount (GBs) of data from Splunk?

rgarcia
Engager

Hello,

I'm trying to export, dump, or download large quantity of data from splunk. So far I tried dump command and the splunk cli search command to do this

-When I ran the search in the UI followed by the dump command and once the search finished I was unable to locate the file. Place I look for was /opt/splunk/var/run/splunk/dispatch, but I may be looking in the wrong system...is it my indexer or searchhead where this file is located?

-using the cli search command created some memory issues or login failures

Other options?

Note:I am the Splunk Admin, 6 indexer, 6 searchheads

Labels (2)
Tags (1)
0 Karma

thambisetty
Super Champion

@rgarcia 

file will be created on the node where you run the search. I believe you run the search on search head so you will see file in search head only.

I found GUI option is best when dumping raw events in GBs.

Below is the query I use to dump logs from Splunk to file:

 

index=foo| eval _dstpath=strftime(_time, "%Y%m%d") | dump basefilename=zscaler

 

_dstpath specifies the directory format under dump directory example below : 20180124

basefilename is the filename starts with under $SPLUNK_HOME/var/run/splunk/dispatch/<sidfrominspectjob>/dump/20180124/zscaler****

use the above search to create a dump for the time range you have chosen : if you choose two days lets say 24 and 25 of Jan, directories will be created like below:

/opt/splunk/var/run/splunk/dispatch/<sidfrominspectjob>/dump/20180124
/opt/splunk/var/run/splunk/dispatch/<sidfrominspectjob>/dump/20180125

make sure that you have enough space on the search head to run search for long time ranges. if you don't have more space on search head then run a search by choosing one day from time picker.

————————————
If this helps, give a like below.
0 Karma

rgarcia
Engager

I tried the GUI and is just not feasible as the export is in txt format (limitations).

Is there a way to just copy (create a backup) or tar existing data locally without having to use queries or cli tools?

0 Karma

rgarcia
Engager

Hi  thambisetty

I tried the dump query and I can see the SID directory, but no dump or file is found after the search is completed (finished with 9 events).

This is the query I'm using index=something sourcetype=something | eval _dstpath=strftime(_time, "%Y%m%d") | dump basefilename=something_something

I will try to export using GUI and see how that works out for me. 

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...