To clarify my question, when you stop Splunk and run a command such as "./splunk show monitor" are you prompted for a user name and then a password?  Is it only when Splunk is running that you are not prompted?  You should be prompted to auth to run that command, regardless if Splunk is running or not.  To me it appears to be an issue where its not necessarily "hanging" but rather it is not displaying the prompt for username and password.  I did see find answers post with the same behavior but no solution yet. https://community.splunk.com/t5/Deployment-Architecture/Splunk-forwarder-on-Linux-splunk-quot-commands-quot-just-hang/td-p/489434 If you add auth to the command does it work? ./splunk list monitor -auth 'admin:password'   If that works, then it likely confirms that you are not getting prompted for credentials.  I can dig into that further and see what I can find.  Thanks! ... View more

A few QQ: 1.  Which OS version are you on?  2.  Do you get a prompt for a username and password when you execute the command?  If not, I assume you are prompted when the service is stopped.   I tested this with Splunk UF 8.0.4 on CentOS 6.4 and 8.2 (as that is what I have loaded in my lab vms) and I did not experience the issue.  I can test on a relevant version of CentOS if it differs from what I have listed above.  Otherwise, review the splunkd.log, splunkd_access.log & audit.log from the times when you attempted the cmd.  Example entries of when it is successful: audit.log: 06-22-2020 11:19:00.117 -0500 INFO AuditLogger - Audit:[timestamp=06-22-2020 11:19:00.117, user=admin, action=login attempt, info=succeeded reason=user-initiated useragent="SplunkCli/6.0 (build 767223ac207f)" clientip=127.0.0.1 session=d082061bc7e043d8d6cb176472acdc85][n/a] 06-22-2020 11:19:00.118 -0500 INFO AuditLogger - Audit:[timestamp=06-22-2020 11:19:00.118, user=admin, action=list_inputs, info=granted ][n/a] 06-22-2020 11:19:00.121 -0500 INFO AuditLogger - Audit:[timestamp=06-22-2020 11:19:00.121, user=admin, action=list_inputs, info=granted ][n/a] 06-22-2020 11:19:00.123 -0500 INFO AuditLogger - Audit:[timestamp=06-22-2020 11:19:00.123, user=admin, action=list_inputs, info=granted object="$SPLUNK_HOME/var/log/splunk" operation=members][n/a] 06-22-2020 11:19:00.126 -0500 INFO AuditLogger - Audit:[timestamp=06-22-2020 11:19:00.126, user=admin, action=list_inputs, info=granted object="$SPLUNK_HOME/var/log/splunk/license_usage_summary.log" operation=members][n/a] 06-22-2020 11:19:00.127 -0500 INFO AuditLogger - Audit:[timestamp=06-22-2020 11:19:00.127, user=admin, action=list_inputs, info=granted object="$SPLUNK_HOME/var/log/splunk/metrics.log" operation=members][n/a] 06-22-2020 11:19:00.128 -0500 INFO AuditLogger - Audit:[timestamp=06-22-2020 11:19:00.128, user=admin, action=list_inputs, info=granted object="$SPLUNK_HOME/var/log/splunk/splunk_instrumentation_cloud.log*" operation=members][n/a] 06-22-2020 11:19:00.129 -0500 INFO AuditLogger - Audit:[timestamp=06-22-2020 11:19:00.129, user=admin, action=list_inputs, info=granted object="$SPLUNK_HOME/var/log/splunk/splunkd.log" operation=members][n/a] 06-22-2020 11:19:00.129 -0500 INFO AuditLogger - Audit:[timestamp=06-22-2020 11:19:00.129, user=admin, action=list_inputs, info=granted object="$SPLUNK_HOME/var/log/watchdog/watchdog.log*" operation=members][n/a] 06-22-2020 11:19:00.130 -0500 INFO AuditLogger - Audit:[timestamp=06-22-2020 11:19:00.130, user=admin, action=list_inputs, info=granted object="$SPLUNK_HOME/var/run/splunk/search_telemetry/*search_telemetry.json" operation=members][n/a] 06-22-2020 11:19:00.131 -0500 INFO AuditLogger - Audit:[timestamp=06-22-2020 11:19:00.131, user=admin, action=list_inputs, info=granted object="$SPLUNK_HOME/var/spool/splunk/...stash_new" operation=members][n/a]  splunkd_access.log 127.0.0.1 - admin [22/Jun/2020:11:19:00.113 -0500] "POST /services/auth/login HTTP/1.1" 200 213 - - - 4ms 127.0.0.1 - admin [22/Jun/2020:11:19:00.118 -0500] "GET /services/data/inputs/monitor/ HTTP/1.1" 200 28945 - - - 2ms 127.0.0.1 - admin [22/Jun/2020:11:19:00.121 -0500] "GET /services/data/inputs/monitor/?count=-1 HTTP/1.1" 200 28963 - - -1ms 127.0.0.1 - admin [22/Jun/2020:11:19:00.123 -0500] "GET /servicesNS/nobody/system/data/inputs/monitor%24SPLUNK_HOME%252Fvar%252Flog%252Fsplunk/members?count=-1 HTTP/1.1" 200 42072 - - - 2ms 127.0.0.1 - admin [22/Jun/2020:11:19:00.126 -0500] "GET /servicesNS/nobody/system/data/inputs/monitor%24SPLUNK_HOME%252Fvar%252Flog%252Fsplunk%252Flicense_usage_summary.log/members?count=-1 HTTP/1.1" 200 4265 - - - 1ms 127.0.0.1 - admin [22/Jun/2020:11:19:00.127 -0500] "GET /servicesNS/nobody/SplunkUniversalForwarder/data/inputs/monitor%24SPLUNK_HOME%252Fvar%252Flog%252Fsplunk%252Fmetrics.log/members?count=-1 HTTP/1.1" 200 4347 - - - 0ms 127.0.0.1 - admin [22/Jun/2020:11:19:00.128 -0500] "GET /servicesNS/nobody/system/data/inputs/monitor%24SPLUNK_HOME%252Fvar%252Flog%252Fsplunk%252Fsplunk_instrumentation_cloud.log%2A/members?count=-1 HTTP/1.1" 200 4314 - - - 1ms 127.0.0.1 - admin [22/Jun/2020:11:19:00.128 -0500] "GET /servicesNS/nobody/SplunkUniversalForwarder/data/inputs/monitor%24SPLUNK_HOME%252Fvar%252Flog%252Fsplunk%252Fsplunkd.log/members?count=-1 HTTP/1.1" 200 4347 - - - 1ms 127.0.0.1 - admin [22/Jun/2020:11:19:00.129 -0500] "GET /servicesNS/nobody/system/data/inputs/monitor%24SPLUNK_HOME%252Fvar%252Flog%252Fwatchdog%252Fwatchdog.log%2A/members?count=-1 HTTP/1.1" 200 4188 - - - 0ms 127.0.0.1 - admin [22/Jun/2020:11:19:00.130 -0500] "GET /servicesNS/nobody/system/data/inputs/monitor%24SPLUNK_HOME%252Fvar%252Frun%252Fsplunk%252Fsearch_telemetry%252F%2Asearch_telemetry.json/members?count=-1 HTTP/1.1" 200 1967 - - - 0ms 127.0.0.1 - admin [22/Jun/2020:11:19:00.131 -0500] "GET /servicesNS/nobody/system/data/inputs/monitor%24SPLUNK_HOME%252Fvar%252Fspool%252Fsplunk%252F...stash_new/members?count=-1 HTTP/1.1" 200 1967 - - - 0ms   You could also run strace against the pid when the cmd is executed. ... View more

Oh gotcha, I'm still getting used to our new community site as well.  Thanks! ... View more

No, you have the new index cluster peers in the data center which are part of the index cluster.  To remove the "old peers" (previously existing index cluster peers), you would have to "offline" them.  From a doc I linked to on my previous post:  https://docs.splunk.com/Documentation/Splunk/8.0.4/Indexer/Takeapeeroffline#Take_a_peer_down_permanently:_the_enforce-counts_offline_command The enforce-counts version of the offline command is intended for use when you want to take a peer offline permanently, but only after the cluster has returned to its complete state. The index cluster itself would still be up and available.  You would simply use splunk offline --enforce-counts On each of the existing peers that you want to remove. ... View more

In serverclass.conf, please include the following at the top level for the Cluster Master's server class: stateOnClient = noop More info on this setting: Spec file: https://docs.splunk.com/Documentation/Splunk/latest/Admin/Serverclassconf stateOnClient = enabled | disabled | noop * If set to "enabled", sets the application state to enabled on the client, regardless of state on the deployment server. * If set to "disabled", set the application state to disabled on the client, regardless of state on the deployment server. * If set to "noop", the state on the client will be the same as on the deployment server. * Can be overridden at the serverClass level and the serverClass:app level. * Defaults to enabled. In your specific example, I would recommend the following settings be used at the top level of the Cluster Master's server class: [serverClass:ClusterMasters] stateOnClient = noop restartSplunkWeb = 0 restartSplunkd = 0 [serverClass:ClusterMasters:app:APPNAME] [serverClass:ClusterMasters:app:APPNAME2] [serverClass:ClusterMasters:app:APPNAME3] [serverClass:ClusterMasters:app:APPNAME4] ... View more

Yes, it is fixed in 7.1.6 and 7.2.4. 7.1.6 is currently available, 7.2.4 should be out in the next couple of weeks as far as I know. ... View more

I apologize, when I was out on holiday it looks like this got moved to 7.2.4. I have updated my answer and put in the bug #. They should be listed in the release notes once its available. If there are further changes I will update it once I am aware. ... View more

I wanted to let you know that the fix for this is currently scheduled for 7.1.5 and 7.2.3. That of course is subject to change but if testing goes well that is when you should see the fix. Thanks! ... View more

Development is actively working on the issue. I do not have a fix version or ETA at this time. As soon as I do, I will provide an update. ... View more

You can add that setting to existing indexes. New buckets will be index cluster buckets and replicate. The previous buckets will remain as standalone buckets, will be searchable, but will not replicate. As an example: DB Directory for an index on an index cluster member, index is not configured with repFactor=auto Oct 4 10:38 db_1538594348_1538594348_0 After adding 'repFactor=auto Oct 4 10:38 db_1538594348_1538594348_0 Oct 4 10:42 db_1538594348_1538594348_1_81D21F8E-64D7-4ACB-962A-3CB331958491 DB Inspect: ... View more

We do not offer single issue patches. You will want to upgrade when its convenient. ... View more

Appears to be the same. For 7.0.x it is fixed in 7.0.6 (SPL-158503). ... View more