We have a distributed environment with a search head cluster. On this search head cluster we have deployed the DB connect app version 3.0.3.
Then we configured all identities and database connections accordingly. Those connections worked fine.
Then we did a rolling restart (may also be caused by a new deployment) of the search head cluster.
Afterwards, the connections to the databases only worked from one member of the search head cluster. On the other members we got the following error when checking the conncection:
Internal server error, originalErrorMessage=Failed to initialize pool: ORA-01017: invalid username/password; logon denied
Just adding this Answer as I suppose there might be others with the same problem. The problem was analyzed together with Splunk Support, Case #461225.
The db connect app encrypts the passwords of identities from version 3.0.0 onwards. The key it uses is saved into the file $SPLUNK_HOME/etc/apps/splunk_app_db_connect/certs/identity.dat
. This file is generated on startup of the splunk instance only if it is inexistent.
That file is inexistent after deploying onto the search head cluster. Therefore each cluster member generates its own key in the file identity.dat. Sadly, that file is not synchronized between the search head cluster members.
So when saving a new identity, the password of that identity is encrypted with the key of the local search head cluster member and written encrypted into the file $SPLUNK_HOME/etc/apps/splunk_app_db_connect/local/identities.conf
.
As each member has its own key, the other members won't be able to decrypt that encrypted password. Hence the username/password mismatch.
-Muryoutaisuu
Just adding this Answer as I suppose there might be others with the same problem. The problem was analyzed together with Splunk Support, Case #461225.
The db connect app encrypts the passwords of identities from version 3.0.0 onwards. The key it uses is saved into the file $SPLUNK_HOME/etc/apps/splunk_app_db_connect/certs/identity.dat
. This file is generated on startup of the splunk instance only if it is inexistent.
That file is inexistent after deploying onto the search head cluster. Therefore each cluster member generates its own key in the file identity.dat. Sadly, that file is not synchronized between the search head cluster members.
So when saving a new identity, the password of that identity is encrypted with the key of the local search head cluster member and written encrypted into the file $SPLUNK_HOME/etc/apps/splunk_app_db_connect/local/identities.conf
.
As each member has its own key, the other members won't be able to decrypt that encrypted password. Hence the username/password mismatch.
-Muryoutaisuu