To clarify my question, when you stop Splunk and run a command such as "./splunk show monitor" are you prompted for a user name and then a password?  Is it only when Splunk is running that you are not prompted?  You should be prompted to auth to run that command, regardless if Splunk is running or not.  To me it appears to be an issue where its not necessarily "hanging" but rather it is not displaying the prompt for username and password.  I did see find answers post with the same behavior but no solution yet. https://community.splunk.com/t5/Deployment-Architecture/Splunk-forwarder-on-Linux-splunk-quot-commands-quot-just-hang/td-p/489434 If you add auth to the command does it work? ./splunk list monitor -auth 'admin:password'   If that works, then it likely confirms that you are not getting prompted for credentials.  I can dig into that further and see what I can find.  Thanks! ... View more

A few QQ: 1.  Which OS version are you on?  2.  Do you get a prompt for a username and password when you execute the command?  If not, I assume you are prompted when the service is stopped.   I tested this with Splunk UF 8.0.4 on CentOS 6.4 and 8.2 (as that is what I have loaded in my lab vms) and I did not experience the issue.  I can test on a relevant version of CentOS if it differs from what I have listed above.  Otherwise, review the splunkd.log, splunkd_access.log & audit.log from the times when you attempted the cmd.  Example entries of when it is successful: audit.log: 06-22-2020 11:19:00.117 -0500 INFO AuditLogger - Audit:[timestamp=06-22-2020 11:19:00.117, user=admin, action=login attempt, info=succeeded reason=user-initiated useragent="SplunkCli/6.0 (build 767223ac207f)" clientip=127.0.0.1 session=d082061bc7e043d8d6cb176472acdc85][n/a] 06-22-2020 11:19:00.118 -0500 INFO AuditLogger - Audit:[timestamp=06-22-2020 11:19:00.118, user=admin, action=list_inputs, info=granted ][n/a] 06-22-2020 11:19:00.121 -0500 INFO AuditLogger - Audit:[timestamp=06-22-2020 11:19:00.121, user=admin, action=list_inputs, info=granted ][n/a] 06-22-2020 11:19:00.123 -0500 INFO AuditLogger - Audit:[timestamp=06-22-2020 11:19:00.123, user=admin, action=list_inputs, info=granted object="$SPLUNK_HOME/var/log/splunk" operation=members][n/a] 06-22-2020 11:19:00.126 -0500 INFO AuditLogger - Audit:[timestamp=06-22-2020 11:19:00.126, user=admin, action=list_inputs, info=granted object="$SPLUNK_HOME/var/log/splunk/license_usage_summary.log" operation=members][n/a] 06-22-2020 11:19:00.127 -0500 INFO AuditLogger - Audit:[timestamp=06-22-2020 11:19:00.127, user=admin, action=list_inputs, info=granted object="$SPLUNK_HOME/var/log/splunk/metrics.log" operation=members][n/a] 06-22-2020 11:19:00.128 -0500 INFO AuditLogger - Audit:[timestamp=06-22-2020 11:19:00.128, user=admin, action=list_inputs, info=granted object="$SPLUNK_HOME/var/log/splunk/splunk_instrumentation_cloud.log*" operation=members][n/a] 06-22-2020 11:19:00.129 -0500 INFO AuditLogger - Audit:[timestamp=06-22-2020 11:19:00.129, user=admin, action=list_inputs, info=granted object="$SPLUNK_HOME/var/log/splunk/splunkd.log" operation=members][n/a] 06-22-2020 11:19:00.129 -0500 INFO AuditLogger - Audit:[timestamp=06-22-2020 11:19:00.129, user=admin, action=list_inputs, info=granted object="$SPLUNK_HOME/var/log/watchdog/watchdog.log*" operation=members][n/a] 06-22-2020 11:19:00.130 -0500 INFO AuditLogger - Audit:[timestamp=06-22-2020 11:19:00.130, user=admin, action=list_inputs, info=granted object="$SPLUNK_HOME/var/run/splunk/search_telemetry/*search_telemetry.json" operation=members][n/a] 06-22-2020 11:19:00.131 -0500 INFO AuditLogger - Audit:[timestamp=06-22-2020 11:19:00.131, user=admin, action=list_inputs, info=granted object="$SPLUNK_HOME/var/spool/splunk/...stash_new" operation=members][n/a]  splunkd_access.log 127.0.0.1 - admin [22/Jun/2020:11:19:00.113 -0500] "POST /services/auth/login HTTP/1.1" 200 213 - - - 4ms 127.0.0.1 - admin [22/Jun/2020:11:19:00.118 -0500] "GET /services/data/inputs/monitor/ HTTP/1.1" 200 28945 - - - 2ms 127.0.0.1 - admin [22/Jun/2020:11:19:00.121 -0500] "GET /services/data/inputs/monitor/?count=-1 HTTP/1.1" 200 28963 - - -1ms 127.0.0.1 - admin [22/Jun/2020:11:19:00.123 -0500] "GET /servicesNS/nobody/system/data/inputs/monitor%24SPLUNK_HOME%252Fvar%252Flog%252Fsplunk/members?count=-1 HTTP/1.1" 200 42072 - - - 2ms 127.0.0.1 - admin [22/Jun/2020:11:19:00.126 -0500] "GET /servicesNS/nobody/system/data/inputs/monitor%24SPLUNK_HOME%252Fvar%252Flog%252Fsplunk%252Flicense_usage_summary.log/members?count=-1 HTTP/1.1" 200 4265 - - - 1ms 127.0.0.1 - admin [22/Jun/2020:11:19:00.127 -0500] "GET /servicesNS/nobody/SplunkUniversalForwarder/data/inputs/monitor%24SPLUNK_HOME%252Fvar%252Flog%252Fsplunk%252Fmetrics.log/members?count=-1 HTTP/1.1" 200 4347 - - - 0ms 127.0.0.1 - admin [22/Jun/2020:11:19:00.128 -0500] "GET /servicesNS/nobody/system/data/inputs/monitor%24SPLUNK_HOME%252Fvar%252Flog%252Fsplunk%252Fsplunk_instrumentation_cloud.log%2A/members?count=-1 HTTP/1.1" 200 4314 - - - 1ms 127.0.0.1 - admin [22/Jun/2020:11:19:00.128 -0500] "GET /servicesNS/nobody/SplunkUniversalForwarder/data/inputs/monitor%24SPLUNK_HOME%252Fvar%252Flog%252Fsplunk%252Fsplunkd.log/members?count=-1 HTTP/1.1" 200 4347 - - - 1ms 127.0.0.1 - admin [22/Jun/2020:11:19:00.129 -0500] "GET /servicesNS/nobody/system/data/inputs/monitor%24SPLUNK_HOME%252Fvar%252Flog%252Fwatchdog%252Fwatchdog.log%2A/members?count=-1 HTTP/1.1" 200 4188 - - - 0ms 127.0.0.1 - admin [22/Jun/2020:11:19:00.130 -0500] "GET /servicesNS/nobody/system/data/inputs/monitor%24SPLUNK_HOME%252Fvar%252Frun%252Fsplunk%252Fsearch_telemetry%252F%2Asearch_telemetry.json/members?count=-1 HTTP/1.1" 200 1967 - - - 0ms 127.0.0.1 - admin [22/Jun/2020:11:19:00.131 -0500] "GET /servicesNS/nobody/system/data/inputs/monitor%24SPLUNK_HOME%252Fvar%252Fspool%252Fsplunk%252F...stash_new/members?count=-1 HTTP/1.1" 200 1967 - - - 0ms   You could also run strace against the pid when the cmd is executed. ... View more

In serverclass.conf, please include the following at the top level for the Cluster Master's server class: stateOnClient = noop More info on this setting: Spec file: https://docs.splunk.com/Documentation/Splunk/latest/Admin/Serverclassconf stateOnClient = enabled | disabled | noop * If set to "enabled", sets the application state to enabled on the client, regardless of state on the deployment server. * If set to "disabled", set the application state to disabled on the client, regardless of state on the deployment server. * If set to "noop", the state on the client will be the same as on the deployment server. * Can be overridden at the serverClass level and the serverClass:app level. * Defaults to enabled. In your specific example, I would recommend the following settings be used at the top level of the Cluster Master's server class: [serverClass:ClusterMasters] stateOnClient = noop restartSplunkWeb = 0 restartSplunkd = 0 [serverClass:ClusterMasters:app:APPNAME] [serverClass:ClusterMasters:app:APPNAME2] [serverClass:ClusterMasters:app:APPNAME3] [serverClass:ClusterMasters:app:APPNAME4] ... View more

You can add that setting to existing indexes. New buckets will be index cluster buckets and replicate. The previous buckets will remain as standalone buckets, will be searchable, but will not replicate. As an example: DB Directory for an index on an index cluster member, index is not configured with repFactor=auto Oct 4 10:38 db_1538594348_1538594348_0 After adding 'repFactor=auto Oct 4 10:38 db_1538594348_1538594348_0 Oct 4 10:42 db_1538594348_1538594348_1_81D21F8E-64D7-4ACB-962A-3CB331958491 DB Inspect: ... View more