Splunk Search

Upper Limit for REST API limits.conf - maxresultrows

Chris_Olson
Splunk Employee
Splunk Employee

Is there an upper end limit on this value? In certain use cases, there might be a need to return a very large number of results.

0 Karma
1 Solution

gkanapathy
Splunk Employee
Splunk Employee

I would not raise the limit. Instead, you can simply make multiple calls to the GET endpoint, in blocks smaller than the default maxresultrows limit of 50,000 until you have exhausted the number of events returned, i.e, the first call uses offset=0&count=50000, the next uses offset=50000&count=50000, then offset=100000&count=50000, etc. Your program that calls the endpoint can output each block as it get it.

View solution in original post

MCW
Loves-to-Learn

Hi experts,

After submitting a search query via REST API, is there a way to check number of events the search results for the job id?

Without which, I won't know if  how many GET each limited to 50K results which is something I run into as well.

Alternatively, is there an argument that I can use in HTTP GET to splunk to override the 50K limit?

Thanks,

MCW

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

I would not raise the limit. Instead, you can simply make multiple calls to the GET endpoint, in blocks smaller than the default maxresultrows limit of 50,000 until you have exhausted the number of events returned, i.e, the first call uses offset=0&count=50000, the next uses offset=50000&count=50000, then offset=100000&count=50000, etc. Your program that calls the endpoint can output each block as it get it.

gkanapathy
Splunk Employee
Splunk Employee

This is incorrect. Splunk lets you query results and events from the REST API before the search has completed. You can see this in effect whenever you perform a large search from the UI (which itself uses the REST API). By trying to engineer smaller searches yourself (which is what you'd do if you were, say, querying against MySQL or a traditional RDBMS, and which is unnecessary in Splunk) you are complicating your code, putting extra load on the server, and possibly preventing your query from effective map-reduce execution.

andras_kerekes
Explorer

One problem with the above is, that Splunk will search over all events, and in case you have a few million events and want to query the first 200000, the search will take rather long time (of course depends on the machine it runs on).

You need to add | head n with appropriate n, e.g. 200000 so that Splunk will return the results as soon as it found the first 200000 events. Further optimization could be to dynamically calculate n e.g. 50k, 100k, 150k, 200k in each respective iteration.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...