All Apps and Add-ons

Sideview Utils: How to change maxout setting in limits.conf

ifeldshteyn
Communicator

Hello,
I can't seem to change maxout setting in limits.conf. I've tried changing every maxout or max_count param I could find but it is not taking effect. I tried changing other variables (like reduce_freq or ttl) and that seemed to work but not maxout or maxtime.

This happens when I run sideview Search of "index=*" and then show results in a table. Here is the search log extract...

06-24-2019 17:51:40.147 INFO  dispatchRunner - Search process mode: preforked (first search in process)
06-24-2019 17:51:40.147 INFO  dispatchRunner - initing LicenseMgr in search process: nonPro=1
06-24-2019 17:51:40.147 INFO  dispatchRunner - registering build time modules, count=1
06-24-2019 17:51:40.147 INFO  dispatchRunner - registering search time components of build time module name=vix
06-24-2019 17:51:40.147 INFO  dispatchRunner - Splunkd starting (build 67571ef4b87d).
06-24-2019 17:51:40.147 INFO  dispatchRunner - System info: Linux, XXX, 3.10.0-957.5.1.el7.x86_64, #1 SMP Wed Dec 19 10:46:58 EST 2018, x86_64.
06-24-2019 17:51:40.148 INFO  dispatchRunner - Detected 32 (virtual) CPUs, 16 CPU cores, and 128731MB RAM
06-24-2019 17:51:40.148 INFO  dispatchRunner - Maximum number of threads (approximate): 16000
06-24-2019 17:51:40.148 INFO  dispatchRunner - Arguments are: "search" "--id=1561413100.2" "--maxbuckets=0" "--ttl=600" "--maxout=10000" "--maxtime=8640000" "--lookups=1" "--reduce_freq=6"
06-24-2019 17:51:40.148 INFO  dispatchRunner - Getting search configuration data from: /opt/splunk/etc/modules/parsing/config.xml
06-24-2019 17:51:40.152 INFO  KVStoreBulletinBoardManager - MessageHandler:KVSTORE_FAILED removed
...<SNIP>...
06-24-2019 17:51:40.477 INFO  DispatchThread - Job truncated due to max_count=10000 reached

In the log above it is limiting results to 10000 .

When I run the search index=_internal on the command line it works perfectly, I can override maxout.

*splunk@XXX:[/opt/splunk/bin]> ./splunk search 'index=_internal' -maxout 10501 | wc -l
10501
*

Here is the btool output....

splunk@xxx:[/opt/splunk/bin]> ./splunk btool limits list --debug | grep maxout
/opt/splunk/etc/system/local/limits.conf subsearch_maxout = 50000
/opt/splunk/etc/system/local/limits.conf maxout = 5000
/opt/splunk/etc/system/local/limits.conf maxout = 5000
splunk@xxx:[/opt/splunk/bin]> ./splunk btool limits list --debug | grep max_count
/opt/splunk/etc/system/local/limits.conf max_count = 10000000
/opt/splunk/etc/system/local/limits.conf alerts_max_count = 50000
/opt/splunk/etc/system/local/limits.conf max_count = 500000
/opt/splunk/etc/system/local/limits.conf max_count = 5000
splunk@xxx:[/opt/splunk/bin]> ./splunk btool limits list --debug | grep 10000
/opt/splunk/etc/system/local/limits.conf max_count = 10000000
/opt/splunk/etc/system/local/limits.conf max_number_of_ack_channel = 1000000
/opt/splunk/etc/system/local/limits.conf max_number_of_acked_requests_pending_query = 10000000
/opt/splunk/etc/system/local/limits.conf max_number_of_acked_requests_pending_query_per_ack_channel = 1000000
/opt/splunk/etc/system/local/limits.conf maxdatapoints = 100000000
/opt/splunk/etc/system/local/limits.conf max_memtable_bytes = 10000000
/opt/splunk/etc/system/local/limits.conf maxcount = 100000
/opt/splunk/etc/system/local/limits.conf batch_search_max_index_values = 10000000
/opt/splunk/etc/system/local/limits.conf batch_search_max_results_aggregator_queue_size = 100000000
/opt/splunk/etc/system/local/limits.conf batch_search_max_serialized_results_queue_size = 100000000
/opt/splunk/etc/system/local/limits.conf max_chunk_queue_size = 10000000
/opt/splunk/etc/system/local/limits.conf max_rawsize_perchunk = 100000000
/opt/splunk/etc/system/local/limits.conf result_queue_max_size = 100000000
/opt/splunk/etc/system/local/limits.conf max_valuemap_bytes = 100000
/opt/splunk/etc/system/local/limits.conf maxopenevents = 100000
/opt/splunk/etc/system/local/limits.conf chunk_size = 10000000

Nothing seems to match 10000. I went through the entire /etc stack and ensured there is no such setting there. Somehow no matter what I do it is picking up this 10000 maxout setting and I have no idea from where. Note, I am using sideview utils - not sure if it matters. Splunk version is 6.5.2 (so it is a bit old) - but this is a self contained UAT instance.

0 Karma
1 Solution

ifeldshteyn
Communicator

I couldn't figure it out so I just hacked the Search.js module by adding

search.setMaxCount(50000)

and that worked (make sure you restart Splunk afterwards).

View solution in original post

0 Karma

ifeldshteyn
Communicator

I couldn't figure it out so I just hacked the Search.js module by adding

search.setMaxCount(50000)

and that worked (make sure you restart Splunk afterwards).

0 Karma

ifeldshteyn
Communicator

Note: It seems that the issue is also present in the latest Splunk release. I've downloaded 7.3.0 and notice the limit of 10000 is still present.

0 Karma
Get Updates on the Splunk Community!

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...