Cluster Master shows error ERROR ApplicationManager - Unexpected duplicate app: _cluster

[rbal_splunk]

On the Cluster master created following inputs.conf in $SPLUNK_HOME/etc/master-apps/_cluster/local/inputs.conf


[SSL]
rootCA = $SPLUNK_HOME/etc/auth/cacert.pem
serverCert = $SPLUNK_HOME/etc/auth/server.pem
password = password


Applied Bundle on Cluster Master
splunk apply cluster-bundle --answer-yes

Once the bundle gets applied on peer- subsequent restart of the cluster peer shows following

[root@beefysup04 bin]# ./splunk restart
ERROR ApplicationManager - Unexpected duplicate app: _cluster
ERROR ApplicationManager - Unexpected duplicate app: _cluster
ERROR ApplicationManager - Unexpected duplicate app: _cluster
ERROR ApplicationManager - Unexpected duplicate app: _cluster
Stopping splunkweb...
                                                           [  OK  ]
Stopping splunkd...
Shutting down.  Please wait, as this may take a few minutes.
.                                                          [  OK  ]
Stopping splunk helpers...
                                                           [  OK  ]
Done.

Splunk> Finding your faults, just like mom.

Checking prerequisites...
ERROR ApplicationManager - Unexpected duplicate app: _cluster
ERROR ApplicationManager - Unexpected duplicate app: _cluster
        Checking http port [24100]: ERROR ApplicationManager - Unexpected duplicate app: _cluster
ERROR ApplicationManager - Unexpected duplicate app: _cluster
open
        Checking mgmt port [24101]: ERROR ApplicationManager - Unexpected duplicate app: _cluster
ERROR ApplicationManager - Unexpected duplicate app: _cluster
open
        Checking configuration...  Done.
        Checking critical directories...        Done
        Checking indexes...
                Validated: _audit _blocksignature _internal _thefishbucket history main summary
        Done
Unexpected duplicate app: _cluster
ERROR ApplicationManager - Unexpected duplicate app: _cluster
ERROR ApplicationManager - Unexpected duplicate app: _cluster
        Checking filesystem compatibility...  Done
ERROR ApplicationManager - Unexpected duplicate app: _cluster
ERROR ApplicationManager - Unexpected duplicate app: _cluster
ERROR ApplicationManager - Unexpected duplicate app: _cluster
ERROR ApplicationManager - Unexpected duplicate app: _cluster
        Checking conf files for typos...        Done
ERROR ApplicationManager - Unexpected duplicate app: _cluster
ERROR ApplicationManager - Unexpected duplicate app: _cluster
        Checking replication_port port [24102]: ERROR ApplicationManager - Unexpected duplicate app: _cluster
ERROR ApplicationManager - Unexpected duplicate app: _cluster
open
ERROR ApplicationManager - Unexpected duplicate app: _cluster
ERROR ApplicationManager - Unexpected duplicate app: _cluster
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
Done
                                                           [  OK  ]
ERROR ApplicationManager - Unexpected duplicate app: _cluster
ERROR ApplicationManager - Unexpected duplicate app: _cluster
ERROR ApplicationManager - Unexpected duplicate app: _cluster
ERROR ApplicationManager - Unexpected duplicate app: _cluster
ERROR ApplicationManager - Unexpected duplicate app: _cluster
ERROR ApplicationManager - Unexpected duplicate app: _cluster
                                                           [  OK  ]
Starting splunkweb...  Done

Is this an issue, Should I be worried.

[rbal_splunk]

The error like above can be ignored, as these warning are caused when one or more App exists both in $SPLUNK_HOME/etc/apps and $SPLUNK_HOME/etc/slave-apps. There may be redundant configuration directives in these locations.

Usually this happen, after app is deployed to on cluster Peer at location $SPLUNK_HOME/etc/slave-apps, which is read-only directory, now the lookup polpulated by app are created at location $SPLUNK_HOME/etc/app. This behavior is being treated by internal Bug SPL-82244.

So the above, messages can be ignored most of the time. Only one exception if is the duplicate app is created due to password hashing as explained below, in that case you need to apply the workaround recommended below

Below is example to explain work around for password hashing workaround

---

Issue :

1) Lets stay On the Cluster master created following inputs.conf

in $SPLUNK_HOME/etc/master-apps/_cluster/local/inputs.conf . the purpose of the file is to send out new password


[SSL]
rootCA = $SPLUNK_HOME/etc/auth/cacert.pem
serverCert = $SPLUNK_HOME/etc/auth/server.pem
password = testpassword


2) Now, Applied Bundle on Cluster Master
splunk apply cluster-bundle --answer-yes

Once the bundle gets applied on peer- subsequent restart of the cluster peer. And you will see messages like “Unexpected duplicate app: _cluster”

3) On the Peer inputs.conf got deployed
in $SPLUNK_HOME/etc/slave-apps/_cluster/local as expected and the content of file is


[SSL]
rootCA = $SPLUNK_HOME/etc/auth/cacert.pem
serverCert = $SPLUNK_HOME/etc/auth/server.pem
password = password


Note that the certificate password is not encrypted at location $SPLUNK_HOME/etc/slave-apps/_cluster/local

In addition, at location $SPLUNK_HOME/etc/apps a new app directory with the name of ‘_cluster’ got created as shown below:

[root@beefysup04 apps]# pwd
/$SPLUNK_HOME/etc/apps
[root@beefysup04 apps]#  ls –l
[root@beefysup04 apps]# ls -l
total 48
drwx------. 4 root root 4096 Mar 23 22:47 _cluster
drwxr-xr-x. 6 root root 4096 Mar 23 21:47 framework
drwxr-xr-x. 6 root root 4096 Feb 11 19:08 gettingstarted
drwxr-xr-x. 6 root root 4096 Feb 11 19:09 launcher
drwxr-xr-x. 5 root root 4096 Mar 23 21:47 learned
drwxr-xr-x. 3 root root 4096 Feb 11 19:09 legacy
drwxr-xr-x. 6 root root 4096 Feb 11 19:08 sample_app
drwxr-xr-x. 9 root root 4096 Feb 11 19:09 search
drwxr-xr-x. 4 root root 4096 Feb 11 19:07 splunk_datapreview
drwxr-xr-x. 4 root root 4096 Feb 11 19:08 SplunkForwarder
drwxr-xr-x. 4 root root 4096 Feb 11 19:08 SplunkLightForwarder
drwxr-xr-x. 4 root root 4096 Feb 11 19:07 user-prefs

For this new app we do see the hashed password in inputs.conf:


cat $SPLUNK_HOME/etc/ apps/_cluster/local/inputs.conf:
[SSL]
password = $1$uoGFCByN8i4x


So the major problem here is that the certificate password is hashed and re-written in a location that is below the clear text password in terms of configuration precedence. From etc/system/local/conf.conf:

(...)
$apps-slave$/*/local = conf
system/local = conf
$apps-base$/*/local = conf
(...)
$apps-slave$/*/default = conf
$apps-base$/*/default = conf
(...)


So, the first thing is: as things are currently being pushed by the customer, they will not have the desired effect because the clear text certificate password will win over the hashed certificate password.

---

Workaround

What I would recommend here is to:
• Create an app in the Cluster Master's etc/master-apps directory dedicated to push inputs.conf to the peers
• Put the customized inputs.conf with the splunktcp SSL input definition in the default directory of that app
As previously observed, on the peers this will result in the hashed certificate password being written to etc/apps//local/inputs.conf, which will win over the clear text password pushed to etc/slave-apps//default/inputs.conf.