All Apps and Add-ons

Splunk DB Connect 2: Identities and Connections not synchronized between SHC members and encrypted passwords in db_connections.conf are not hashed using splunk.secret

darrend
Path Finder

Hi Guys

Issue and Resolution

We recently encountered an interesting issue with Search Head Clustering when using it with Splunk DB Connect version 2.

We wanted to push the DBXv2 app from our deployer to our cluster members, but we wanted to manage the identities and connections using 1 of the SHC members and have them sync to the other members rather than either managing the connections from the deployer instance, or having to configure the settings on each member node.

In our scenario we are also sharing a common splunk.secret across our SHC members.

Issue 1 - The identities and connections were not synchronized between cluster members.

Firstly we deployed the app as per the instructions, and attempted to configure an identity and connection via an SHC member. This worked on the first node, but did not sync the content to the other members.

Solution:

On the deployer we added the following configuration to the local\server.conf that was pushed from the deployer in the splunk_app_for_dbconnect app.

[shclustering]
conf_replication_include.identities = true
conf_replication_include.db_connections = true

These two configuration items control the synchronisation of custom config items and are not enabled by default, there is a reference in the apps default\server.conf to a conf_replication_include.* parameter, but this does not seem to work as intended so we explicitly defined the files.

We then redeployed the app from the deployer.

Once it had pushed the app we created a new identity and connection in the UI and that was successfully synchronised to the other member nodes, but it highlighted the second issue below.

Issue 2 - the encrypted passwords in db_connections.conf are not hashed using splunk.secret as the seed

After the db_connections.conf was synced between nodes we noticed that the password hash was the same in each file as we would expect with a shared splunk.secret, however the authentication to the database server only seemed to function on the node that we originally configured the identity. This behaviour did not meet our requirements so we investigated further.

Solution:

After some investigation it became apparent that the Splunk App For DB Connect V2 does not use the splunk.secret to encrypt the passwords in db_connections.conf

We found that DBXv2 actually uses a seed file inside the certs folder of the app called identity.dat

To solve the password issue we copied the identity.dat file from 1 of our SHC members to the same location on the app to be deployed from the deployer, we then redeployed the app.

Once the app had been redeployed successfully we checked the identity.dat on each node to ensure it was the same, this essentially makes the identity.dat behave the same as a shared splunk.secret

From here all we needed to do is edit the identities on any of the SHC member nodes and retyped the password, and select save.

Success you can now access the database from any member node and only have to manage the identities and connections on which ever member node you happen to be connected to.

Thanks
Darren

1 Solution

darrend
Path Finder

Solution to Issue 1:

On the deployer we added the following configuration to the local\server.conf that was pushed from the deployer in the splunk_app_for_dbconnect app.

 [shclustering]
 conf_replication_include.identities = true
 conf_replication_include.db_connections = true

These two configuration items control the synchronisation of custom config items and are not enabled by default, there is a reference in the apps default\server.conf to a conf_replication_include.* parameter, but this does not seem to work as intended so we explicitly defined the files.

We then redeployed the app from the deployer.

Once it had pushed the app we created a new identity and connection in the UI and that was successfully synchronised to the other member nodes, but it highlighted the second issue below.

Solution to Issue 2:

After some investigation it became apparent that the Splunk App For DB Connect V2 does not use the splunk.secret to encrypt the passwords in db_connections.conf

We found that DBXv2 actually uses a seed file inside the certs folder of the app called identity.dat

To solve the password issue we copied the identity.dat file from 1 of our SHC members to the same location on the app to be deployed from the deployer, we then redeployed the app.

Once the app had been redeployed successfully we checked the identity.dat on each node to ensure it was the same, this essentially makes the identity.dat behave the same as a shared splunk.secret

From here all we needed to do is edit the identities on any of the SHC member nodes and retyped the password, and select save.

Success you can now access the database from any member node and only have to manage the identities and connections on which ever member node you happen to be connected to.

View solution in original post

darrend
Path Finder

Solution to Issue 1:

On the deployer we added the following configuration to the local\server.conf that was pushed from the deployer in the splunk_app_for_dbconnect app.

 [shclustering]
 conf_replication_include.identities = true
 conf_replication_include.db_connections = true

These two configuration items control the synchronisation of custom config items and are not enabled by default, there is a reference in the apps default\server.conf to a conf_replication_include.* parameter, but this does not seem to work as intended so we explicitly defined the files.

We then redeployed the app from the deployer.

Once it had pushed the app we created a new identity and connection in the UI and that was successfully synchronised to the other member nodes, but it highlighted the second issue below.

Solution to Issue 2:

After some investigation it became apparent that the Splunk App For DB Connect V2 does not use the splunk.secret to encrypt the passwords in db_connections.conf

We found that DBXv2 actually uses a seed file inside the certs folder of the app called identity.dat

To solve the password issue we copied the identity.dat file from 1 of our SHC members to the same location on the app to be deployed from the deployer, we then redeployed the app.

Once the app had been redeployed successfully we checked the identity.dat on each node to ensure it was the same, this essentially makes the identity.dat behave the same as a shared splunk.secret

From here all we needed to do is edit the identities on any of the SHC member nodes and retyped the password, and select save.

Success you can now access the database from any member node and only have to manage the identities and connections on which ever member node you happen to be connected to.

christian_clout
Explorer

I am also experiencing issues like these using a standard deployment server. However in my case, using the same identities.conf (with encrypted password) and identity.dat does not result in a working configuration once deployed.

0 Karma

a212830
Champion

I am having the same problem. When I looked at this app on the deployer, there was no local directory (but there was a locale directory). I created the local directory, and created a server.conf with those settings and then executed a shcluster apply, but it doesn't look like the local directory was pushed out. Are these the same steps that you followed?

0 Karma

ppablo
Retired

Hi @darrend

Thanks for sharing your issues and solutions with the Splunk Answers community. Can you actually post your solutions as an actual answer below in the "Enter your answer here..." box? Otherwise, this post will float around displayed as having no answer and will remain unresolved. This will help for other users who are actually searching for an answer to the same problem. I'll be sure to upvote the answer ones you get that posted.

Patrick

0 Karma

jdastmalchi_spl
Splunk Employee
Splunk Employee

The first issue will be addressed in DB connect 2.1 as DBX-2108.
For the second issue there have been improvements to this area but this may still be the correct approach as if you follow the documentation it recommends to nitially set up some identity and connection then deploy the app which means you will automatically deploy identity.dat

0 Karma

splunkreal
Motivator

Hello, this is really painful, Splunk apps should be 100% cluster compliant and sync all files without needing deployer setup 🙂. Thanks.

* If this helps, please upvote or accept solution 🙂 *
0 Karma

darrend
Path Finder

Thanks for the feedback Jaleh 🙂

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...