Why does Splunk Web limit the total of number of rows in a result that can be exported to a CSV file?

[reed_kelly]

Does anyone know the technical or another reason that the Splunk Web interface limits the total number of rows of the result set that can be exported to a CSV file? I know that unlimited sizes can be achieved by the REST API or SDK, but I have users who want to extract huge amounts of data. Since they are not handy with the command line, I end up using curl on their behalf.

[nabeel652]

As far as I have seen it doesn't limit the rows, or maybe I am getting you wrong. However, I think the problem may be in the roles. If you assign a user role to someone his search size is limited to 100MB and therefore it stops once the search occupies more than 100MB. Change "Limit Total Jobs Disk Quota" in the roles settings and this may fix the problem

[abhijit_mhatre · Accepted solution]

The default limit for csv export from a saved search is 10000. However, if you want to change this:

1) You can go to savedsearches.conf & change it.

savedsearches.conf
action.email.maxresults = <integer>
* Set the maximum number of results which you want to export
* Currently defaults to 10000

2) You can export data using outputlookup command which will create a csv file in Splunk. You can export unlimited number of results from the lookup file now. Use this at the end of your query.

For example:
index=abc | search host=* | stats count by b | outputlookup hosts.csv

This will create a lookup file hosts.csv & you can export it by running | inputlookup hosts.csv

Let me know if it helps.

[reed_kelly]

I should have clarified. I am talking about the export to file option on the main search interface. This allows the end user to export directly to their desktop. I suppose exporting to a lookup file would allow this by giving the users access to the Lookup Editor, but that could also increase our search bundle and cause issues with distributed search bundle replication.

There is a maxresultrows=nnn in limits.conf, but the documentation explicitly says not to set this higher than 50000.

My question is Why? And if there is a technical reason, then is there an app that would run the search in the REST API and then let the user download the result >100000000 rows and/or >100GB, etc...

[abhijit_mhatre]

There is app "Splunk for Excel Export" which will help you to download 1 million events per workbook. Please check the app compatibility with your Splunk version.

App link : https://splunkbase.splunk.com/app/760/#/details
Splunk for Excel Export

Thanks & Regards,
Abhijit Mhatre