Sideview Utils: How to change maxout setting in limits.conf

[ifeldshteyn]

Hello,
I can't seem to change maxout setting in limits.conf. I've tried changing every maxout or max_count param I could find but it is not taking effect. I tried changing other variables (like reduce_freq or ttl) and that seemed to work but not maxout or maxtime.

This happens when I run sideview Search of "index=*" and then show results in a table. Here is the search log extract...

06-24-2019 17:51:40.147 INFO  dispatchRunner - Search process mode: preforked (first search in process)
06-24-2019 17:51:40.147 INFO  dispatchRunner - initing LicenseMgr in search process: nonPro=1
06-24-2019 17:51:40.147 INFO  dispatchRunner - registering build time modules, count=1
06-24-2019 17:51:40.147 INFO  dispatchRunner - registering search time components of build time module name=vix
06-24-2019 17:51:40.147 INFO  dispatchRunner - Splunkd starting (build 67571ef4b87d).
06-24-2019 17:51:40.147 INFO  dispatchRunner - System info: Linux, XXX, 3.10.0-957.5.1.el7.x86_64, #1 SMP Wed Dec 19 10:46:58 EST 2018, x86_64.
06-24-2019 17:51:40.148 INFO  dispatchRunner - Detected 32 (virtual) CPUs, 16 CPU cores, and 128731MB RAM
06-24-2019 17:51:40.148 INFO  dispatchRunner - Maximum number of threads (approximate): 16000
06-24-2019 17:51:40.148 INFO  dispatchRunner - Arguments are: "search" "--id=1561413100.2" "--maxbuckets=0" "--ttl=600" "--maxout=10000" "--maxtime=8640000" "--lookups=1" "--reduce_freq=6"
06-24-2019 17:51:40.148 INFO  dispatchRunner - Getting search configuration data from: /opt/splunk/etc/modules/parsing/config.xml
06-24-2019 17:51:40.152 INFO  KVStoreBulletinBoardManager - MessageHandler:KVSTORE_FAILED removed
...<SNIP>...
06-24-2019 17:51:40.477 INFO  DispatchThread - Job truncated due to max_count=10000 reached

In the log above it is limiting results to 10000 .

When I run the search index=_internal on the command line it works perfectly, I can override maxout.

*splunk@XXX:[/opt/splunk/bin]> ./splunk search 'index=_internal' -maxout 10501 | wc -l
10501
*

Here is the btool output....

splunk@xxx:[/opt/splunk/bin]> ./splunk btool limits list --debug | grep maxout
/opt/splunk/etc/system/local/limits.conf subsearch_maxout = 50000
/opt/splunk/etc/system/local/limits.conf maxout = 5000
/opt/splunk/etc/system/local/limits.conf maxout = 5000
splunk@xxx:[/opt/splunk/bin]> ./splunk btool limits list --debug | grep max_count
/opt/splunk/etc/system/local/limits.conf max_count = 10000000
/opt/splunk/etc/system/local/limits.conf alerts_max_count = 50000
/opt/splunk/etc/system/local/limits.conf max_count = 500000
/opt/splunk/etc/system/local/limits.conf max_count = 5000
splunk@xxx:[/opt/splunk/bin]> ./splunk btool limits list --debug | grep 10000
/opt/splunk/etc/system/local/limits.conf max_count = 10000000
/opt/splunk/etc/system/local/limits.conf max_number_of_ack_channel = 1000000
/opt/splunk/etc/system/local/limits.conf max_number_of_acked_requests_pending_query = 10000000
/opt/splunk/etc/system/local/limits.conf max_number_of_acked_requests_pending_query_per_ack_channel = 1000000
/opt/splunk/etc/system/local/limits.conf maxdatapoints = 100000000
/opt/splunk/etc/system/local/limits.conf max_memtable_bytes = 10000000
/opt/splunk/etc/system/local/limits.conf maxcount = 100000
/opt/splunk/etc/system/local/limits.conf batch_search_max_index_values = 10000000
/opt/splunk/etc/system/local/limits.conf batch_search_max_results_aggregator_queue_size = 100000000
/opt/splunk/etc/system/local/limits.conf batch_search_max_serialized_results_queue_size = 100000000
/opt/splunk/etc/system/local/limits.conf max_chunk_queue_size = 10000000
/opt/splunk/etc/system/local/limits.conf max_rawsize_perchunk = 100000000
/opt/splunk/etc/system/local/limits.conf result_queue_max_size = 100000000
/opt/splunk/etc/system/local/limits.conf max_valuemap_bytes = 100000
/opt/splunk/etc/system/local/limits.conf maxopenevents = 100000
/opt/splunk/etc/system/local/limits.conf chunk_size = 10000000

Nothing seems to match 10000. I went through the entire /etc stack and ensured there is no such setting there. Somehow no matter what I do it is picking up this 10000 maxout setting and I have no idea from where. Note, I am using sideview utils - not sure if it matters. Splunk version is 6.5.2 (so it is a bit old) - but this is a self contained UAT instance.

[ifeldshteyn · Accepted solution]

I couldn't figure it out so I just hacked the Search.js module by adding

search.setMaxCount(50000)

and that worked (make sure you restart Splunk afterwards).

[ifeldshteyn]

Note: It seems that the issue is also present in the latest Splunk release. I've downloaded 7.3.0 and notice the limit of 10000 is still present.