Exporting CSV over 10,000 No OS access

[carmackd]

I’m looking for a solution to export a 100,000+ row csv file without giving out OS level access to our search head (outputcsv). Some of our splunk users are involved with collecting large amounts of data for legal cases. They need quick access to their results, but we cannot give them OS level access. I’m aware of the work around that breaks your outputcsv up into 10,000 row segments so you can export them through the UI, but this method is cumbersome, and leaves a mess of csv files behind.
http://blogs.splunk.com/2009/08/07/help-i-cant-export-more-than-10000-events

Does the splunk UI have the ability to access the file system and extract the files created by outputcsv in $SPLUNK_HOME/var/run/splunk/? If not, would it be possible to build a user interface within a splunk app to access the file system?

I’m open to any suggestions, but like the idea of a UI solution.

[mmletzko]

We have a script that's executed after the search is done that SCPs the csv file to a Windows NT file server and then deletes the CSV on the Splunk Server (Solaris).

This gets the file to the user without them having to have access to the Splunk Server's OS.

[shirolu]

| outputlookup youcsv.csv
no limits

[the_wolverine]

outputcsv also work just fine after removing the sort command -- export from UI is no longer capped at 10k.

[DanielFordWA]

Quick note - When I use a sort command outputcsv is limited to 10,000. Don't know why but it works fine without sort.

[sideview · Accepted solution]

1) create one saved search for each csv (that is just | inputcsv filename) and if they run the saved search they'll at least get taken to the search UI where they can sort and filter the data in the csv.

2) create one saved search for each csv, and also create a single custom form search view. That view gives them the option of picking a saved search in a pulldown. where these guys pick which saved search they want, (which amounts to picking the csv) and then the UI could them some simple controls to sort, page or even report on the data in that csv...

and if they can report on it such that the report has <10,000 rows we can throw an export button into that interface too.

If you're pretty familiar with the advanced XML you could take a stab at it, or (pls forgive this if it seems like a plug) you could hire a splunk consultant (like me) to knock it out.

3) If the number of csv's we're talking about is rather large or if it's just a PITA to create a saved search for each of them.... or if they need to be generated on a schedule and automatically named ( http://answers.splunk.com/questions/10552/dynamic-naming-of-files-with-outputcsv ), then it's still possible but it's a different kettle of fish and would require a little custom splunk development.