Dashboards & Visualizations

Export more than 50k records in Advanced XML using the Export module

oded4478
Explorer

I've added an ability to an Advanced XML report to export search results using the Export module.
Exporting is by default limited to 50k results.

Can this limit be set to 100k in the limits.conf file?
I tried different settings but nothing seems to do the trick.

Tags (2)
1 Solution

sideview
SplunkTrust
SplunkTrust

Splunk's Export module just turns around and hits a REST API endpoint on splunkd. In particular it hits /servicesNS///search/jobs//results/export

The Export module does not pass a count param to this endpoint although the endpoint does accept one. Regardless, if you were to pass count=100000 to the REST endpoint, the behavior is still restricted by the following stanza in limits.conf

[restapi]
maxresultrows = 50000

Which limits the maximum number of returned rows presumably across the entire search API.

(Note that this is not a limit on number of rows that are piped around between different commands in a report - just the number of rows you can get out using the REST endpoints .)

Anyway, changing that limits.conf to 100,000 seems to work fine, in that I'm able to hit the endpoint directly and download a csv containing all rows of a 90,000 row search result.

However I urge strong caution as limits are not placed in limits.conf without good reason. Quite possibly or even probably, setting this value too high can result in seriously degraded performance or instability on the server. If I had to guess you might be able to send the system into swap because 50,000 rows is approximately the safe limit of how many rows you can fit into memory at once on minimum spec hardware.

You might also take a look at the Splunk For Excel Export app, which claims to be able to export millions of events.

http://splunk-base.splunk.com/apps/29336/splunk-for-excel-export
I believe the app actually does the export in a streaming fashion and does not use the export endpoint at all so it might somehow not be subject to this limit.

View solution in original post

sideview
SplunkTrust
SplunkTrust

Splunk's Export module just turns around and hits a REST API endpoint on splunkd. In particular it hits /servicesNS///search/jobs//results/export

The Export module does not pass a count param to this endpoint although the endpoint does accept one. Regardless, if you were to pass count=100000 to the REST endpoint, the behavior is still restricted by the following stanza in limits.conf

[restapi]
maxresultrows = 50000

Which limits the maximum number of returned rows presumably across the entire search API.

(Note that this is not a limit on number of rows that are piped around between different commands in a report - just the number of rows you can get out using the REST endpoints .)

Anyway, changing that limits.conf to 100,000 seems to work fine, in that I'm able to hit the endpoint directly and download a csv containing all rows of a 90,000 row search result.

However I urge strong caution as limits are not placed in limits.conf without good reason. Quite possibly or even probably, setting this value too high can result in seriously degraded performance or instability on the server. If I had to guess you might be able to send the system into swap because 50,000 rows is approximately the safe limit of how many rows you can fit into memory at once on minimum spec hardware.

You might also take a look at the Splunk For Excel Export app, which claims to be able to export millions of events.

http://splunk-base.splunk.com/apps/29336/splunk-for-excel-export
I believe the app actually does the export in a streaming fashion and does not use the export endpoint at all so it might somehow not be subject to this limit.

oded4478
Explorer

Great, that worked perfect!
I can confirm the Export module can now export more than 50k records.

I will also take a look at the Excel Export module, but a few months ago I ran into a problem with it not handling chinese charsets well.
But that is something for another post.

0 Karma

cramasta
Builder

what version are you running?

0 Karma
Get Updates on the Splunk Community!

The Great Resilience Quest: 10th Leaderboard Update

The tenth leaderboard update (11.23-12.05) for The Great Resilience Quest is out >> As our brave ...

Customer Experience | Call for Stories: Your 2023 Journey with Splunk!

Share your Splunk journey: Splunk is committed to supporting our customers toward success. As the year draws ...

Infographic provides the TL;DR for the 2023 Splunk Career Impact Report

We’ve been shouting it from the rooftops! The findings from the 2023 Splunk Career Impact Report showing that ...