Splunk Search

100 result limit in JS SDK

LordVoldemort
Explorer

I tried adding "count" to params object when calling service.search() but it doesn't work. How do I get more than 100 results from the api when using the javascript SDK?

Tags (3)
1 Solution

ineeman
Splunk Employee
Splunk Employee

Great question. There are a couple of different limits on how to get more data from a Splunk search, but I'll only go over the important ones here.

When you create a search, you can specify how many status buckets this search can have, and how many events/results should be stored per status bucket. The default is 10,000 per status bucket, and the default number of status buckets is 0. So if you did a search with:

service.search("query", function(...) { ... })

The maximum number of results you can get back is 10,000. If you instead did:

service.search("query", {status_buckets: 300}, function(...) { ... })

You would get a maximum of 300 * 10,000 results stored. Note that 300 is the default value used by the UI. You could also pass in something like:

{max_count: 50000}

and that will similarly change things.

However, even with setting max_count/status_buckets, the default number of results you get back when you do something like:

mySearchJob.results({}, function(err, results) { ... })

is going to be 100, per the REST API documentation (here). You can pass in a count parameter to override this, as such:

mySearchJob.results({count: 500}, function(err, results) { ... })

and this will fetch 500 results at a time. You can also paginate through the results by using a combination of count and offset, e.g.:

mySearchJob.results({count: 500, offset: 1000}, function(err, results) { ... })

Finally, the maximum number of results you can get in a single call to the search/jobs/{sid}/results endpoint (which is what mySearchJob.results(...) is calling under the hood) is defined by the maxresultsrows value in limits.conf (in the restapi stanza). This is by default 50,000. Note that you can retrieve more than 50,000 results from a single search, you just have to do it using multiple API calls (i.e. pagination). You can see a sample of this here.

Hopefully this makes sense. If not, let me know, and I'll clarify some more! 🙂

View solution in original post

ineeman
Splunk Employee
Splunk Employee

Great question. There are a couple of different limits on how to get more data from a Splunk search, but I'll only go over the important ones here.

When you create a search, you can specify how many status buckets this search can have, and how many events/results should be stored per status bucket. The default is 10,000 per status bucket, and the default number of status buckets is 0. So if you did a search with:

service.search("query", function(...) { ... })

The maximum number of results you can get back is 10,000. If you instead did:

service.search("query", {status_buckets: 300}, function(...) { ... })

You would get a maximum of 300 * 10,000 results stored. Note that 300 is the default value used by the UI. You could also pass in something like:

{max_count: 50000}

and that will similarly change things.

However, even with setting max_count/status_buckets, the default number of results you get back when you do something like:

mySearchJob.results({}, function(err, results) { ... })

is going to be 100, per the REST API documentation (here). You can pass in a count parameter to override this, as such:

mySearchJob.results({count: 500}, function(err, results) { ... })

and this will fetch 500 results at a time. You can also paginate through the results by using a combination of count and offset, e.g.:

mySearchJob.results({count: 500, offset: 1000}, function(err, results) { ... })

Finally, the maximum number of results you can get in a single call to the search/jobs/{sid}/results endpoint (which is what mySearchJob.results(...) is calling under the hood) is defined by the maxresultsrows value in limits.conf (in the restapi stanza). This is by default 50,000. Note that you can retrieve more than 50,000 results from a single search, you just have to do it using multiple API calls (i.e. pagination). You can see a sample of this here.

Hopefully this makes sense. If not, let me know, and I'll clarify some more! 🙂

sideview
SplunkTrust
SplunkTrust

I think these comments about status buckets only apply if the search has no transforming commands. Status buckets are relevant really only to the streaming portion of the search. Just fyi.

0 Karma

LordVoldemort
Explorer

Thanks for the thorough response.

In this case, putting count in the call to job.results() was all I needed.

0 Karma

ChrisG
Splunk Employee
Splunk Employee

I am far from expert on this. But I can quote some information I have from an expert:

The maximum number of results is always going to be the value in max_count (which you can set when you create the search). The maximum number of events is going to be max_count * status_buckets, which you can also set when you create the search.

Get Updates on the Splunk Community!

Index This | Divide 100 by half. What do you get?

November 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Stay Connected: Your Guide to December Tech Talks, Office Hours, and Webinars!

❄️ Celebrate the season with our December lineup of Community Office Hours, Tech Talks, and Webinars! ...

Splunk and Fraud

Watch Now!Watch an insightful webinar where we delve into the innovative approaches to solving fraud using the ...