How to export > 10000 Events to a .csv via CLI with good performance?

[tpaulsen]

Hello,

in Splunk 3 we were exporting during night time via cronjob 1-hour chunks of data from the previous day via CLI. All together more than 800000 events of data. We need the export because Splunk can´t visualize certain aspects of the data in a report.

In Splunk 3 each 1-hour chunk export took about 8 - 10 minutes to export.

In Splunk 4 the same 1-hour export takes literally HOURS to export, though one export is only about 120 - 250,000 events. Within the Splunk GUI the search takes about 1 minute to run.

Here´s the CLI command for Splunk 4:

./splunk search 'index="idx_prod_online" host="blade504" source="/var/opt/noa/prod/current/online/log/online1.http.log" | regex _raw!="^#"' -earliest_time -1h@h -latest_time @h -maxout 0 -auth username:password >> /tmp/LogEventsRaw/splunkexport.log

What can we do to speed up the command?

The |outputcsv option from here http://blogs.splunk.com/2009/08/07/help-i-cant-export-more-than-10000-events/ doesn´t work in our case, since the diskspace for the searchhead is very limited.

Please help.

[tpaulsen]

We are importing the splunk-export into "CIC tool" a special "Intershop" tool to visualize certain aspects of our business. We already consulted the Splunk support about it, and they admitted that Splunk can´t do this special kind of visualization.

The problem is simply that with Splunk 3 the bulk export worked fine, but with Splunk 4 we have problems to get the data out.

[Johnvey]

Would you mind elaborating on what kind of reporting you are attempting to do? In general, bulk exporting raw events from Splunk is a method of last resort.