Archive
Highlighted

Dump Command - How to Save Files on Remote Path and Question About maxlocal

Explorer

We're trying to export data out of a very large splunk index using the dump command into multiple csv files where the min file size is 1GB of each file.

E.g here's our command ...|dump basefilename=netezza rollsize=1024 format=csv

What we want is to have the files saved on a remote location not local on the Splunk server. Is there a way to do this?

Also, what does the maxlocal parameter mean? We're a little confused by that. It says once it reaches 1GB it moves it to HDFS? We don't have HDFS configured, what happens once it reaches 1GB? Is there a way to make maxlocal unlimited?

Tags (1)
Highlighted

Re: Dump Command - How to Save Files on Remote Path and Question About maxlocal

Communicator

Did you ever come across a way to do this or get it to work correctly.

I could see this being useful in an incident response investigation where we needed to give a large data set out to vendor or 3rd party for analysis rather than searching within splunk.

0 Karma
Highlighted

Re: Dump Command - How to Save Files on Remote Path and Question About maxlocal

Path Finder

@steverimar were you able to figure this out?
I am using Splunk 6.4.2. With dump command, when I provide maxlocal argument, it complains and returns error 'invalid argument' (not using HDFS). Did you find a way to save the output in remote system?

Highlighted

Re: Dump Command - How to Save Files on Remote Path and Question About maxlocal

Splunk Employee
Splunk Employee

The maxlocal argument never worked properly and has been removed from the code and the documentation.

0 Karma
Highlighted

Re: Dump Command - How to Save Files on Remote Path and Question About maxlocal

Contributor

Did you get this resolved?
I've done something similar, there are a couple of ways to get the data onto a remote server.
In the past I have setup a SSHFS mount at $SPLUNK_HOME/var/run/splunk/dispatch, note that this location will be used for all searches.
Stop Splunk before mounting the directory, you'll need to temporarily move out the old directory in order to mount it correctly. Check the file permissions are correct and then start Splunk.
The output of your DUMP commands will now be in $SPLUNK_HOME/var/run/splunk/dispatch/<sid>/dump/ where is the ID of your search.
It isnt the most elegant solution but it worked for me.

Hope this helps!

0 Karma