How do I extend the number of results that an external script returns to more than 100000 lines?

[andrewtrobec]

Hello,

I have an external script that makes calculations. The problem is that it is limiting the number of results to 100000. By default it is 50000, but I managed to extend it to 100000 by adding the following stanzas to limits.conf under the app's local folder:

 [searchresults] 
 maxresultrows = 100000
 [stats]
 maxresultrows = 100000
 [top]
 maxresultrows = 100000

Now I'd like to extend that limit to 500000, but updating the maxresultrows values does not make any difference. For reference, my limits.conf file now looks like this:

[default]
max_mem_usage_mb = 0    
[searchresults]
maxresultrows = 500000    
[stats]
maxresultrows = 500000    
[top]
maxresultrows = 500000    
[set]
maxresultrows = 500000    
[anomalousvalue]
maxresultrows = 500000

What am I missing?

Thank you and best regards,

Andrew

[andrewtrobec · Accepted solution]

I was able to discover that limits were being set in the system local folder which were taking precedence. After removing them, I was able to solve my issue. Useful link for those with similar problems: http://docs.splunk.com/Documentation/Splunk/6.6.4/Admin/Wheretofindtheconfigurationfiles

[efavreau]

Are you looking to view/export all those events, or perform some commands to them? Having your search and/or more detail, would help in getting us the answer you're looking for. One of your comments mentions looking in the inspector, so I suspect you're in the GUI. Have you tried to use the REST API to get all the events?

There's max_count parameter "for searches returning more than the default maximum of 10000 events. Otherwise you may not be able to retrieve results in excess of the default."
Doc: http://docs.splunk.com/Documentation/Splunk/7.1.3/RESTTUT/RESTsearches

[andrewtrobec]

Thanks for the input. I've not tried the REST approach, but I will keep it in my toolkit for sure. In the meantime I was able to discover a limit being set in the system local folder.

[tkopchak]

It isn't recommended to increase the default maxresultrows in the [searchresults] stanza of limits.conf per the documentation:

• This limit should not exceed 50000. Setting this limit higher than 50000 causes instability.

Reference: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf

It's possible that another configuration could be in play here, can you post the search you are running?

[andrewtrobec]

Thanks for the reply. You were correct about the other configuration: I found a limit set in the system local folder which took precedence!

[felipesewaybric]

Can you please post the search?

[andrewtrobec]

Thanks for your input. I was able to resolve by removing some limits in the system local folder.

[woodcock]

I suspect that you are not seeing the entire picture because that setting does not directly control what you are trying to do. It would very, VERY much help to see the search that you are using (because that is likely where the limit is hitting) as well as some sample data.

[andrewtrobec]

Thanks for replying.

After doing some deep diving into the filesystem I discovered that some limits were being set in the system local directory, which takes precedence over the app local directory: http://docs.splunk.com/Documentation/Splunk/6.6.4/Admin/Wheretofindtheconfigurationfiles

After updating that file I was able to resolve the problem!

[mayurr98]

have a look at these two answers:
https://answers.splunk.com/answers/122872/what-is-the-effect-of-maxresultrows-for-stats-in-limits-co...

https://answers.splunk.com/answers/117714/local-limits-conf-not-working-for-the-specified-span-would...

[andrewtrobec]

Thanks for providing the links, but unfortunately they do not help. When I inspect the job, i see the following line:

09-13-2018 20:45:56.066 INFO  script - Invoked script calculate with 16894672 input bytes (100000 events).  Returned 16319858 output bytes in 10767 ms.

It seems that 100000 events are being passed to the script (named calculate). Not sure whether that helps.