Getting Data In

Exporting CSV over 10,000 No OS access

carmackd
Communicator

I’m looking for a solution to export a 100,000+ row csv file without giving out OS level access to our search head (outputcsv). Some of our splunk users are involved with collecting large amounts of data for legal cases. They need quick access to their results, but we cannot give them OS level access. I’m aware of the work around that breaks your outputcsv up into 10,000 row segments so you can export them through the UI, but this method is cumbersome, and leaves a mess of csv files behind.
http://blogs.splunk.com/2009/08/07/help-i-cant-export-more-than-10000-events

Does the splunk UI have the ability to access the file system and extract the files created by outputcsv in $SPLUNK_HOME/var/run/splunk/? If not, would it be possible to build a user interface within a splunk app to access the file system?

I’m open to any suggestions, but like the idea of a UI solution.

Tags (4)
1 Solution

sideview
SplunkTrust
SplunkTrust

1) create one saved search for each csv (that is just | inputcsv filename) and if they run the saved search they'll at least get taken to the search UI where they can sort and filter the data in the csv.

2) create one saved search for each csv, and also create a single custom form search view. That view gives them the option of picking a saved search in a pulldown. where these guys pick which saved search they want, (which amounts to picking the csv) and then the UI could them some simple controls to sort, page or even report on the data in that csv...

and if they can report on it such that the report has <10,000 rows we can throw an export button into that interface too.

If you're pretty familiar with the advanced XML you could take a stab at it, or (pls forgive this if it seems like a plug) you could hire a splunk consultant (like me) to knock it out.

3) If the number of csv's we're talking about is rather large or if it's just a PITA to create a saved search for each of them.... or if they need to be generated on a schedule and automatically named ( http://answers.splunk.com/questions/10552/dynamic-naming-of-files-with-outputcsv ), then it's still possible but it's a different kettle of fish and would require a little custom splunk development.

View solution in original post

mmletzko
Path Finder

We have a script that's executed after the search is done that SCPs the csv file to a Windows NT file server and then deletes the CSV on the Splunk Server (Solaris).

This gets the file to the user without them having to have access to the Splunk Server's OS.

shirolu
Explorer

| outputlookup youcsv.csv
no limits

the_wolverine
Champion

outputcsv also work just fine after removing the sort command -- export from UI is no longer capped at 10k.

0 Karma

DanielFordWA
Contributor

Quick note - When I use a sort command outputcsv is limited to 10,000. Don't know why but it works fine without sort.

sideview
SplunkTrust
SplunkTrust

1) create one saved search for each csv (that is just | inputcsv filename) and if they run the saved search they'll at least get taken to the search UI where they can sort and filter the data in the csv.

2) create one saved search for each csv, and also create a single custom form search view. That view gives them the option of picking a saved search in a pulldown. where these guys pick which saved search they want, (which amounts to picking the csv) and then the UI could them some simple controls to sort, page or even report on the data in that csv...

and if they can report on it such that the report has <10,000 rows we can throw an export button into that interface too.

If you're pretty familiar with the advanced XML you could take a stab at it, or (pls forgive this if it seems like a plug) you could hire a splunk consultant (like me) to knock it out.

3) If the number of csv's we're talking about is rather large or if it's just a PITA to create a saved search for each of them.... or if they need to be generated on a schedule and automatically named ( http://answers.splunk.com/questions/10552/dynamic-naming-of-files-with-outputcsv ), then it's still possible but it's a different kettle of fish and would require a little custom splunk development.

Get Updates on the Splunk Community!

New Splunk Observability innovations: Deeper visibility and smarter alerting to ...

You asked, we delivered. Splunk Observability Cloud has several new innovations giving you deeper visibility ...

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...