All Apps and Add-ons

db connect app on searchhead cluster, username/password mismatch after rolling restart

Communicator

We have a distributed environment with a search head cluster. On this search head cluster we have deployed the DB connect app version 3.0.3.
Then we configured all identities and database connections accordingly. Those connections worked fine.
Then we did a rolling restart (may also be caused by a new deployment) of the search head cluster.
Afterwards, the connections to the databases only worked from one member of the search head cluster. On the other members we got the following error when checking the conncection:

Internal server error, originalErrorMessage=Failed to initialize pool: ORA-01017: invalid username/password; logon denied 
0 Karma
1 Solution

Communicator

Just adding this Answer as I suppose there might be others with the same problem. The problem was analyzed together with Splunk Support, Case #461225.

The db connect app encrypts the passwords of identities from version 3.0.0 onwards. The key it uses is saved into the file $SPLUNK_HOME/etc/apps/splunk_app_db_connect/certs/identity.dat. This file is generated on startup of the splunk instance only if it is inexistent.
That file is inexistent after deploying onto the search head cluster. Therefore each cluster member generates its own key in the file identity.dat. Sadly, that file is not synchronized between the search head cluster members.
So when saving a new identity, the password of that identity is encrypted with the key of the local search head cluster member and written encrypted into the file $SPLUNK_HOME/etc/apps/splunk_app_db_connect/local/identities.conf.
As each member has its own key, the other members won't be able to decrypt that encrypted password. Hence the username/password mismatch.

Workaround:

  1. create app on deployer & restart deployer
  2. copy $SPLUNKHOME/etc/apps/splunkappdbconnect/certs/identity.dat to $SPLUNKHOME/etc/shcluster/apps/splunkappdbconnect/certs/identity.dat
  3. deploy onto searchhead cluster via deployer
  4. configure the db connection on the searchhead cluster
  5. check connection on non-captain member -> worked
  6. do a rolling restart of the searchhead cluster
  7. check connection on non-captain member -> worked

-Muryoutaisuu

View solution in original post

Communicator

Just adding this Answer as I suppose there might be others with the same problem. The problem was analyzed together with Splunk Support, Case #461225.

The db connect app encrypts the passwords of identities from version 3.0.0 onwards. The key it uses is saved into the file $SPLUNK_HOME/etc/apps/splunk_app_db_connect/certs/identity.dat. This file is generated on startup of the splunk instance only if it is inexistent.
That file is inexistent after deploying onto the search head cluster. Therefore each cluster member generates its own key in the file identity.dat. Sadly, that file is not synchronized between the search head cluster members.
So when saving a new identity, the password of that identity is encrypted with the key of the local search head cluster member and written encrypted into the file $SPLUNK_HOME/etc/apps/splunk_app_db_connect/local/identities.conf.
As each member has its own key, the other members won't be able to decrypt that encrypted password. Hence the username/password mismatch.

Workaround:

  1. create app on deployer & restart deployer
  2. copy $SPLUNKHOME/etc/apps/splunkappdbconnect/certs/identity.dat to $SPLUNKHOME/etc/shcluster/apps/splunkappdbconnect/certs/identity.dat
  3. deploy onto searchhead cluster via deployer
  4. configure the db connection on the searchhead cluster
  5. check connection on non-captain member -> worked
  6. do a rolling restart of the searchhead cluster
  7. check connection on non-captain member -> worked

-Muryoutaisuu

View solution in original post