Splunk Dev

How do I extend the number of results that an external script returns to more than 100000 lines?

andrewtrobec
Motivator

Hello,

I have an external script that makes calculations. The problem is that it is limiting the number of results to 100000. By default it is 50000, but I managed to extend it to 100000 by adding the following stanzas to limits.conf under the app's local folder:

 [searchresults] 
 maxresultrows = 100000
 [stats]
 maxresultrows = 100000
 [top]
 maxresultrows = 100000

Now I'd like to extend that limit to 500000, but updating the maxresultrows values does not make any difference. For reference, my limits.conf file now looks like this:

[default]
max_mem_usage_mb = 0    
[searchresults]
maxresultrows = 500000    
[stats]
maxresultrows = 500000    
[top]
maxresultrows = 500000    
[set]
maxresultrows = 500000    
[anomalousvalue]
maxresultrows = 500000

What am I missing?

Thank you and best regards,

Andrew

Tags (1)
0 Karma
1 Solution

andrewtrobec
Motivator

I was able to discover that limits were being set in the system local folder which were taking precedence. After removing them, I was able to solve my issue. Useful link for those with similar problems: http://docs.splunk.com/Documentation/Splunk/6.6.4/Admin/Wheretofindtheconfigurationfiles

View solution in original post

0 Karma

andrewtrobec
Motivator

I was able to discover that limits were being set in the system local folder which were taking precedence. After removing them, I was able to solve my issue. Useful link for those with similar problems: http://docs.splunk.com/Documentation/Splunk/6.6.4/Admin/Wheretofindtheconfigurationfiles

0 Karma

efavreau
Motivator

Are you looking to view/export all those events, or perform some commands to them? Having your search and/or more detail, would help in getting us the answer you're looking for. One of your comments mentions looking in the inspector, so I suspect you're in the GUI. Have you tried to use the REST API to get all the events?

There's max_count parameter "for searches returning more than the default maximum of 10000 events. Otherwise you may not be able to retrieve results in excess of the default."
Doc: http://docs.splunk.com/Documentation/Splunk/7.1.3/RESTTUT/RESTsearches

###

If this reply helps you, an upvote would be appreciated.

andrewtrobec
Motivator

Thanks for the input. I've not tried the REST approach, but I will keep it in my toolkit for sure. In the meantime I was able to discover a limit being set in the system local folder.

0 Karma

tkopchak
Path Finder

It isn't recommended to increase the default maxresultrows in the [searchresults] stanza of limits.conf per the documentation:

  • This limit should not exceed 50000. Setting this limit higher than 50000 causes instability.

Reference: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf

It's possible that another configuration could be in play here, can you post the search you are running?

andrewtrobec
Motivator

Thanks for the reply. You were correct about the other configuration: I found a limit set in the system local folder which took precedence!

0 Karma

felipesewaybric
Contributor

Can you please post the search?

andrewtrobec
Motivator

Thanks for your input. I was able to resolve by removing some limits in the system local folder.

0 Karma

woodcock
Esteemed Legend

I suspect that you are not seeing the entire picture because that setting does not directly control what you are trying to do. It would very, VERY much help to see the search that you are using (because that is likely where the limit is hitting) as well as some sample data.

andrewtrobec
Motivator

Thanks for replying.

After doing some deep diving into the filesystem I discovered that some limits were being set in the system local directory, which takes precedence over the app local directory: http://docs.splunk.com/Documentation/Splunk/6.6.4/Admin/Wheretofindtheconfigurationfiles

After updating that file I was able to resolve the problem!

0 Karma

andrewtrobec
Motivator

Thanks for providing the links, but unfortunately they do not help. When I inspect the job, i see the following line:

09-13-2018 20:45:56.066 INFO  script - Invoked script calculate with 16894672 input bytes (100000 events).  Returned 16319858 output bytes in 10767 ms.

It seems that 100000 events are being passed to the script (named calculate). Not sure whether that helps.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...