db connect app on searchhead cluster, username/password mismatch after rolling restart

[Muryoutaisuu]

We have a distributed environment with a search head cluster. On this search head cluster we have deployed the DB connect app version 3.0.3.
Then we configured all identities and database connections accordingly. Those connections worked fine.
Then we did a rolling restart (may also be caused by a new deployment) of the search head cluster.
Afterwards, the connections to the databases only worked from one member of the search head cluster. On the other members we got the following error when checking the conncection:

Internal server error, originalErrorMessage=Failed to initialize pool: ORA-01017: invalid username/password; logon denied 

[Muryoutaisuu · Accepted solution]

Just adding this Answer as I suppose there might be others with the same problem. The problem was analyzed together with Splunk Support, Case #461225.

The db connect app encrypts the passwords of identities from version 3.0.0 onwards. The key it uses is saved into the file $SPLUNK_HOME/etc/apps/splunk_app_db_connect/certs/identity.dat. This file is generated on startup of the splunk instance only if it is inexistent.
That file is inexistent after deploying onto the search head cluster. Therefore each cluster member generates its own key in the file identity.dat. Sadly, that file is not synchronized between the search head cluster members.
So when saving a new identity, the password of that identity is encrypted with the key of the local search head cluster member and written encrypted into the file $SPLUNK_HOME/etc/apps/splunk_app_db_connect/local/identities.conf.
As each member has its own key, the other members won't be able to decrypt that encrypted password. Hence the username/password mismatch.

Workaround:

1. create app on deployer & restart deployer
2. copy $SPLUNK_HOME/etc/apps/splunk_app_db_connect/certs/identity.dat to $SPLUNK_HOME/etc/shcluster/apps/splunk_app_db_connect/certs/identity.dat
3. deploy onto searchhead cluster via deployer
4. configure the db connection on the searchhead cluster
5. check connection on non-captain member -> worked
6. do a rolling restart of the searchhead cluster
7. check connection on non-captain member -> worked

-Muryoutaisuu