topic Re: Dump Command - How to Save Files on Remote Path and Question About maxlocal in Splunk Search

Dump Command - How to Save Files on Remote Path and Question About maxlocal

steverimar — Fri, 15 May 2015 17:15:41 GMT

We're trying to export data out of a very large splunk index using the dump command into multiple csv files where the min file size is 1GB of each file.

E.g here's our command ...|dump basefilename=netezza rollsize=1024 format=csv

What we want is to have the files saved on a remote location not local on the Splunk server. Is there a way to do this?

Also, what does the maxlocal parameter mean? We're a little confused by that. It says once it reaches 1GB it moves it to HDFS? We don't have HDFS configured, what happens once it reaches 1GB? Is there a way to make maxlocal unlimited?

Re: Dump Command - How to Save Files on Remote Path and Question About maxlocal

millern4 — Mon, 14 Sep 2015 12:12:56 GMT

Did you ever come across a way to do this or get it to work correctly.

I could see this being useful in an incident response investigation where we needed to give a large data set out to vendor or 3rd party for analysis rather than searching within splunk.

Re: Dump Command - How to Save Files on Remote Path and Question About maxlocal

kausar — Wed, 03 Aug 2016 19:31:29 GMT

@steverimar were you able to figure this out?
I am using Splunk 6.4.2. With dump command, when I provide maxlocal argument, it complains and returns error 'invalid argument' (not using HDFS). Did you find a way to save the output in remote system?

Re: Dump Command - How to Save Files on Remote Path and Question About maxlocal

lstewart_splunk — Thu, 03 Aug 2017 15:53:47 GMT

The maxlocal argument never worked properly and has been removed from the code and the documentation.

Re: Dump Command - How to Save Files on Remote Path and Question About maxlocal

livehybrid — Mon, 05 Mar 2018 10:35:14 GMT

Did you get this resolved?
I've done something similar, there are a couple of ways to get the data onto a remote server.
In the past I have setup a SSHFS mount at $SPLUNK_HOME/var/run/splunk/dispatch, note that this location will be used for all searches.
Stop Splunk before mounting the directory, you'll need to temporarily move out the old directory in order to mount it correctly. Check the file permissions are correct and then start Splunk.
The output of your DUMP commands will now be in $SPLUNK_HOME/var/run/splunk/dispatch/<sid>/dump/ where is the ID of your search.
It isnt the most elegant solution but it worked for me.

Hope this helps!