Getting Data In

Exporting CSV over 10,000 No OS access

carmackd
Communicator

I’m looking for a solution to export a 100,000+ row csv file without giving out OS level access to our search head (outputcsv). Some of our splunk users are involved with collecting large amounts of data for legal cases. They need quick access to their results, but we cannot give them OS level access. I’m aware of the work around that breaks your outputcsv up into 10,000 row segments so you can export them through the UI, but this method is cumbersome, and leaves a mess of csv files behind.
http://blogs.splunk.com/2009/08/07/help-i-cant-export-more-than-10000-events

Does the splunk UI have the ability to access the file system and extract the files created by outputcsv in $SPLUNK_HOME/var/run/splunk/? If not, would it be possible to build a user interface within a splunk app to access the file system?

I’m open to any suggestions, but like the idea of a UI solution.

Tags (4)
1 Solution

sideview
SplunkTrust
SplunkTrust

1) create one saved search for each csv (that is just | inputcsv filename) and if they run the saved search they'll at least get taken to the search UI where they can sort and filter the data in the csv.

2) create one saved search for each csv, and also create a single custom form search view. That view gives them the option of picking a saved search in a pulldown. where these guys pick which saved search they want, (which amounts to picking the csv) and then the UI could them some simple controls to sort, page or even report on the data in that csv...

and if they can report on it such that the report has <10,000 rows we can throw an export button into that interface too.

If you're pretty familiar with the advanced XML you could take a stab at it, or (pls forgive this if it seems like a plug) you could hire a splunk consultant (like me) to knock it out.

3) If the number of csv's we're talking about is rather large or if it's just a PITA to create a saved search for each of them.... or if they need to be generated on a schedule and automatically named ( http://answers.splunk.com/questions/10552/dynamic-naming-of-files-with-outputcsv ), then it's still possible but it's a different kettle of fish and would require a little custom splunk development.

View solution in original post

mmletzko
Path Finder

We have a script that's executed after the search is done that SCPs the csv file to a Windows NT file server and then deletes the CSV on the Splunk Server (Solaris).

This gets the file to the user without them having to have access to the Splunk Server's OS.

shirolu
Explorer

| outputlookup youcsv.csv
no limits

the_wolverine
Champion

outputcsv also work just fine after removing the sort command -- export from UI is no longer capped at 10k.

0 Karma

DanielFordWA
Contributor

Quick note - When I use a sort command outputcsv is limited to 10,000. Don't know why but it works fine without sort.

sideview
SplunkTrust
SplunkTrust

1) create one saved search for each csv (that is just | inputcsv filename) and if they run the saved search they'll at least get taken to the search UI where they can sort and filter the data in the csv.

2) create one saved search for each csv, and also create a single custom form search view. That view gives them the option of picking a saved search in a pulldown. where these guys pick which saved search they want, (which amounts to picking the csv) and then the UI could them some simple controls to sort, page or even report on the data in that csv...

and if they can report on it such that the report has <10,000 rows we can throw an export button into that interface too.

If you're pretty familiar with the advanced XML you could take a stab at it, or (pls forgive this if it seems like a plug) you could hire a splunk consultant (like me) to knock it out.

3) If the number of csv's we're talking about is rather large or if it's just a PITA to create a saved search for each of them.... or if they need to be generated on a schedule and automatically named ( http://answers.splunk.com/questions/10552/dynamic-naming-of-files-with-outputcsv ), then it's still possible but it's a different kettle of fish and would require a little custom splunk development.

Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...