The error like above can be ignored, as these warning are caused when one or more App exists both in $SPLUNK_HOME/etc/apps and $SPLUNK_HOME/etc/slave-apps. There may be redundant configuration directives in these locations.
Usually this happen, after app is deployed to on cluster Peer at location $SPLUNK_HOME/etc/slave-apps, which is read-only directory, now the lookup polpulated by app are created at location $SPLUNK_HOME/etc/app. This behavior is being treated by internal Bug SPL-82244.
So the above, messages can be ignored most of the time. Only one exception if is the duplicate app is created due to password hashing as explained below, in that case you need to apply the workaround recommended below
Below is example to explain work around for password hashing workaround
Issue :
1) Lets stay On the Cluster master created following inputs.conf
in $SPLUNK_HOME/etc/master-apps/_cluster/local/inputs.conf . the purpose of the file is to send out new password
[SSL]
rootCA = $SPLUNK_HOME/etc/auth/cacert.pem
serverCert = $SPLUNK_HOME/etc/auth/server.pem
password = testpassword
2) Now, Applied Bundle on Cluster Master
splunk apply cluster-bundle --answer-yes
Once the bundle gets applied on peer- subsequent restart of the cluster peer. And you will see messages like “Unexpected duplicate app: _cluster”
3) On the Peer inputs.conf got deployed
in $SPLUNK_HOME/etc/slave-apps/_cluster/local as expected and the content of file is
[SSL]
rootCA = $SPLUNK_HOME/etc/auth/cacert.pem
serverCert = $SPLUNK_HOME/etc/auth/server.pem
password = password
Note that the certificate password is not encrypted at location $SPLUNK_HOME/etc/slave-apps/_cluster/local
In addition, at location $SPLUNK_HOME/etc/apps a new app directory with the name of ‘_cluster’ got created as shown below:
[root@beefysup04 apps]# pwd
/$SPLUNK_HOME/etc/apps
[root@beefysup04 apps]# ls –l
[root@beefysup04 apps]# ls -l
total 48
drwx------. 4 root root 4096 Mar 23 22:47 _cluster
drwxr-xr-x. 6 root root 4096 Mar 23 21:47 framework
drwxr-xr-x. 6 root root 4096 Feb 11 19:08 gettingstarted
drwxr-xr-x. 6 root root 4096 Feb 11 19:09 launcher
drwxr-xr-x. 5 root root 4096 Mar 23 21:47 learned
drwxr-xr-x. 3 root root 4096 Feb 11 19:09 legacy
drwxr-xr-x. 6 root root 4096 Feb 11 19:08 sample_app
drwxr-xr-x. 9 root root 4096 Feb 11 19:09 search
drwxr-xr-x. 4 root root 4096 Feb 11 19:07 splunk_datapreview
drwxr-xr-x. 4 root root 4096 Feb 11 19:08 SplunkForwarder
drwxr-xr-x. 4 root root 4096 Feb 11 19:08 SplunkLightForwarder
drwxr-xr-x. 4 root root 4096 Feb 11 19:07 user-prefs
For this new app we do see the hashed password in inputs.conf:
cat $SPLUNK_HOME/etc/ apps/_cluster/local/inputs.conf:
[SSL]
password = $1$uoGFCByN8i4x
So the major problem here is that the certificate password is hashed and re-written in a location that is below the clear text password in terms of configuration precedence. From etc/system/local/conf.conf:
(...)
$apps-slave$/*/local = conf
system/local = conf
$apps-base$/*/local = conf
(...)
$apps-slave$/*/default = conf
$apps-base$/*/default = conf
(...)
So, the first thing is: as things are currently being pushed by the customer, they will not have the desired effect because the clear text certificate password will win over the hashed certificate password.
Workaround
What I would recommend here is to:
• Create an app in the Cluster Master's etc/master-apps directory dedicated to push inputs.conf to the peers
• Put the customized inputs.conf with the splunktcp SSL input definition in the default directory of that app
As previously observed, on the peers this will result in the hashed certificate password being written to etc/apps/ /local/inputs.conf, which will win over the clear text password pushed to etc/slave-apps/ /default/inputs.conf.
... View more
