Hi Guys
Issue and Resolution
We recently encountered an interesting issue with Search Head Clustering when using it with Splunk DB Connect version 2.
We wanted to push the DBXv2 app from our deployer to our cluster members, but we wanted to manage the identities and connections using 1 of the SHC members and have them sync to the other members rather than either managing the connections from the deployer instance, or having to configure the settings on each member node.
In our scenario we are also sharing a common splunk.secret across our SHC members.
Issue 1 - The identities and connections were not synchronized between cluster members.
Firstly we deployed the app as per the instructions, and attempted to configure an identity and connection via an SHC member. This worked on the first node, but did not sync the content to the other members.
Solution:
On the deployer we added the following configuration to the local\server.conf that was pushed from the deployer in the splunk_app_for_dbconnect app.
[shclustering]
conf_replication_include.identities = true
conf_replication_include.db_connections = true
These two configuration items control the synchronisation of custom config items and are not enabled by default, there is a reference in the apps default\server.conf to a conf_replication_include.* parameter, but this does not seem to work as intended so we explicitly defined the files.
We then redeployed the app from the deployer.
Once it had pushed the app we created a new identity and connection in the UI and that was successfully synchronised to the other member nodes, but it highlighted the second issue below.
Issue 2 - the encrypted passwords in db_connections.conf are not hashed using splunk.secret as the seed
After the db_connections.conf was synced between nodes we noticed that the password hash was the same in each file as we would expect with a shared splunk.secret , however the authentication to the database server only seemed to function on the node that we originally configured the identity. This behaviour did not meet our requirements so we investigated further.
Solution:
After some investigation it became apparent that the Splunk App For DB Connect V2 does not use the splunk.secret to encrypt the passwords in db_connections.conf
We found that DBXv2 actually uses a seed file inside the certs folder of the app called identity.dat
To solve the password issue we copied the identity.dat file from 1 of our SHC members to the same location on the app to be deployed from the deployer, we then redeployed the app.
Once the app had been redeployed successfully we checked the identity.dat on each node to ensure it was the same, this essentially makes the identity.dat behave the same as a shared splunk.secret
From here all we needed to do is edit the identities on any of the SHC member nodes and retyped the password, and select save.
Success you can now access the database from any member node and only have to manage the identities and connections on which ever member node you happen to be connected to.
Thanks
Darren
... View more