Issues With DB Connect identities.conf on Search Head Cluster

[dstuder]

I'm trying to use DB Connect on our search heads to do something like this ...

| dbxquery query="My Query" connection="My_Connection"

This sorta works, but only on one search head. The issue seems to be when the identities.conf file syncs to the other heads the encrypted password is not readable by the other instances. It works on the machine that the SQL identity was created on but no others. So, I'm thinking I either need to somehow get identities.conf to not sync and manually create it on each search head or the other search heads need to be able to read the encrypted password. Or maybe there is another solution I'm not thinking of. Anybody have any thoughts with this? Thanks.

[somesoni2]

Splunk SHC would not replicate DB Connect Identities and connection between them by default. 

To resolve this, on the deployer add added the following configuration to the local\server.conf that was pushed from the deployer in the splunk_app_for_dbconnect app.

[shclustering]
 conf_replication_include.identities = true
 conf_replication_include.db_connections = true

 This will make sure that any new identity you create from one SHC member will get replicated to other members.

You can also copy the encrypted identity from that one SH where it exists and deploy it to all SH using SHC deployer.

[dstuder]

Those are in the default/server.conf file and they do sync across the search heads already. The issue is that the identity encrypted password is only readable on the search head that it was created on through the UI.

[sushi]

Did you use deployer for installing db connect on search head clusters?
If you didn't use, you may need to configure with deployer.

First, install db connect on deployer and configure apps.
Next, copy db connect dir to $SPLUNK_HOME/etc/shcluster/apps/ dir.
Finally, deploy the configuration bundle by running the splunk apply and rolling restart shc.

For details, please read splunk document.

https://docs.splunk.com/Documentation/DBX/3.6.0/DeployDBX/Distributeddeployment#Deploy_DB_Connect_on... 

[dstuder]

I did deploy it to the search heads from the search head deployer. However, the SQL identity was created through the UI and then it synced across to the other search heads. So, the SQL connections and the identities do not exist on the search head deployer.

[sushi]

Have you configured db apps identities on deployer before deploy to shc?

[dstuder]

Also, I don't think you can configure the identities at the command line because the password attribute is an encrypted value. I tried putting in plain text and restarting Splunk and it did not encrypt the value on it's own, so I think it has to be done through the UI. I wonder if I need to set ...

conf_replication_include.identities = false

in local/server.conf on the search head deployer, deploy it, and then create the identity on each search head manually.

https://docs.splunk.com/Documentation/DBX/3.7.0/DeployDBX/javaspec

[dstuder]

No, I created the identity through the UI. So, it was synced across the search heads. It didn't get deployed from the search head deployer.

[sushi · Accepted solution]

As document, you should configure following steps.

1. Install db connect apps on deployer.

2. Set up identities and connections for your database on deployer from web UI.

3. Copy the splunk_app_db_connect directory from $SPLUNK_HOME/etc/apps/ to the $SPLUNK_HOME/etc/shcluster/apps/ directory on the deployer. 

4. Deploy the configuration bundle by running the splunk apply shcluster-bundle command on the deployer.

Note that kerberos_client.conf, identity.dat cannot be replicated to other SHC nodes after it's been modified. You need to copy the files manually to other SHC nodes.

[dstuder]

This indeed did fix the issue. Thanks for the help.

[dstuder]

Oh, I missed that in the docs. Thanks. I'll give that a go. I gotta say Splunk really should make a way to manage all these distributed configs in an easier fashion. It's so backwards.

[493669]

@dstuder Not sure on search head cluster but I would prefer to use db-connect on heavy forwarder where there won't be any password sync issue. If you can move db-connect app to Heavy forwarder would solve this issue

[dstuder]

We do have it on the heavy forwarder for indexed data, but on the search heads this would be for search time things such as alerts and reports so alas I do still need it on the search heads as well. My users don't log into anything but the search heads.