Archive
Highlighted

How do I extend the number of results that an external script returns to more than 100000 lines?

Builder

Hello,

I have an external script that makes calculations. The problem is that it is limiting the number of results to 100000. By default it is 50000, but I managed to extend it to 100000 by adding the following stanzas to limits.conf under the app's local folder:

 [searchresults] 
 maxresultrows = 100000
 [stats]
 maxresultrows = 100000
 [top]
 maxresultrows = 100000

Now I'd like to extend that limit to 500000, but updating the maxresultrows values does not make any difference. For reference, my limits.conf file now looks like this:

[default]
max_mem_usage_mb = 0    
[searchresults]
maxresultrows = 500000    
[stats]
maxresultrows = 500000    
[top]
maxresultrows = 500000    
[set]
maxresultrows = 500000    
[anomalousvalue]
maxresultrows = 500000

What am I missing?

Thank you and best regards,

Andrew

Tags (1)
0 Karma
Highlighted

Re: How do I extend the number of results that an external script returns to more than 100000 lines?

SplunkTrust
SplunkTrust
Highlighted

Re: How do I extend the number of results that an external script returns to more than 100000 lines?

Builder

Thanks for providing the links, but unfortunately they do not help. When I inspect the job, i see the following line:

09-13-2018 20:45:56.066 INFO  script - Invoked script calculate with 16894672 input bytes (100000 events).  Returned 16319858 output bytes in 10767 ms.

It seems that 100000 events are being passed to the script (named calculate). Not sure whether that helps.

0 Karma
Highlighted

Re: How do I extend the number of results that an external script returns to more than 100000 lines?

Esteemed Legend

I suspect that you are not seeing the entire picture because that setting does not directly control what you are trying to do. It would very, VERY much help to see the search that you are using (because that is likely where the limit is hitting) as well as some sample data.

Highlighted

Re: How do I extend the number of results that an external script returns to more than 100000 lines?

Builder

Thanks for replying.

After doing some deep diving into the filesystem I discovered that some limits were being set in the system local directory, which takes precedence over the app local directory: http://docs.splunk.com/Documentation/Splunk/6.6.4/Admin/Wheretofindtheconfigurationfiles

After updating that file I was able to resolve the problem!

0 Karma
Highlighted

Re: How do I extend the number of results that an external script returns to more than 100000 lines?

Contributor

Can you please post the search?

Highlighted

Re: How do I extend the number of results that an external script returns to more than 100000 lines?

Builder

Thanks for your input. I was able to resolve by removing some limits in the system local folder.

0 Karma
Highlighted

Re: How do I extend the number of results that an external script returns to more than 100000 lines?

SplunkTrust
SplunkTrust

It isn't recommended to increase the default maxresultrows in the [searchresults] stanza of limits.conf per the documentation:

  • This limit should not exceed 50000. Setting this limit higher than 50000 causes instability.

Reference: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf

It's possible that another configuration could be in play here, can you post the search you are running?

Highlighted

Re: How do I extend the number of results that an external script returns to more than 100000 lines?

Builder

Thanks for the reply. You were correct about the other configuration: I found a limit set in the system local folder which took precedence!

0 Karma
Highlighted

Re: How do I extend the number of results that an external script returns to more than 100000 lines?

Builder

Are you looking to view/export all those events, or perform some commands to them? Having your search and/or more detail, would help in getting us the answer you're looking for. One of your comments mentions looking in the inspector, so I suspect you're in the GUI. Have you tried to use the REST API to get all the events?

There's max_count parameter "for searches returning more than the default maximum of 10000 events. Otherwise you may not be able to retrieve results in excess of the default."
Doc: http://docs.splunk.com/Documentation/Splunk/7.1.3/RESTTUT/RESTsearches

###

If this reply helps you, an upvote would be appreciated.