It is possible to increase maxresultrows over 50000 even if documentation says that it is unstable?

[rajiv_abraham]

------ Start of Edit -------------
EDIT 1: Use Case
- The production server is sends analytics events to Splunk as tagged, log entries. I have a Python script which runs every 5 mins and searches for specific analytics tags within the logs and ingests that into our data warehouse. This incremental search is now htting the 50k limit

EDIT 2: Python Code

 def search(self, text, options):
    search_string = 'search ' + text
    kwargs_oneshot = {"earliest_time": options['start_time'],
                      "latest_time": options['end_time']}

    oneshotsearch_results = self.service.jobs.oneshot(search_string, **kwargs_oneshot)
    lazy_results = results.ResultsReader(oneshotsearch_results)

    return [l for l in lazy_results]

-------- End of Edit -------------
Hi,
In my use case, I can't lose data during search. I'm currently hitting the max limit of 50000 result rows. I am currently reducing the time interval for my searches but our log data is scaling up. The documentation says the following about maxresultrows.

maxresultrows =
* This limit should not exceed 50000. Setting this limit higher than 50000

causes instability.

• Other posts in this forum (can't post links, low karma) talk about increasing the limit to 100000 for e.g. But according to the above documentation, won't we lose results?
  
• If we do lose results if we go over 50k, how can I get above that? Is there some kind of paging mechanism which allows me to go above this limit without losing data?

[sideview]

You should definitely post the SPL syntax for the search that is hitting the limit. Much or most of the time folks in the community can advise you on a different way to get the same end results that will not be subject to the limits at all.

[skoelpin]

Fair enough, I'm interested in seeing what he's trying to accomplish

[skoelpin]

I had this same problem, after 50k rows it will truncate the results based on a FIFO. You can increase this value in the limits.conf and you will retain all results up to that value

[rajiv_abraham]

thanks @skoelpin, so you found the documentation to be wrong about the instability after 50k with your testing?

[skoelpin]

I set the value to be 500,000 and have not noticed a difference on my setup. The instability the docs are referring to are memory.. If the search process exceeds max_mem_usage_mb and maxresultrows then data will spill over to the disk. I would try increasing this value then monitor your memory usage for a little bit to verify its good