Deployment Architecture

Issues With DB Connect identities.conf on Search Head Cluster

dstuder
Communicator

I'm trying to use DB Connect on our search heads to do something like this ...

 

| dbxquery query="My Query" connection="My_Connection"

 

This sorta works, but only on one search head. The issue seems to be when the identities.conf file syncs to the other heads the encrypted password is not readable by the other instances. It works on the machine that the SQL identity was created on but no others. So, I'm thinking I either need to somehow get identities.conf to not sync and manually create it on each search head or the other search heads need to be able to read the encrypted password. Or maybe there is another solution I'm not thinking of. Anybody have any thoughts with this? Thanks.

Labels (1)
Tags (1)
1 Solution

sushi
Explorer

As document, you should configure following steps.

1. Install db connect apps on deployer.

2. Set up identities and connections for your database on deployer from web UI.

3. Copy the splunk_app_db_connect directory from $SPLUNK_HOME/etc/apps/ to the $SPLUNK_HOME/etc/shcluster/apps/ directory on the deployer. 

4. Deploy the configuration bundle by running the splunk apply shcluster-bundle command on the deployer.

 

Note that kerberos_client.conf, identity.dat cannot be replicated to other SHC nodes after it's been modified. You need to copy the files manually to other SHC nodes.

View solution in original post

0 Karma

somesoni2
Revered Legend

Splunk SHC would not replicate DB Connect Identities and connection between them by default. 

To resolve this, on the deployer add added the following configuration to the local\server.conf that was pushed from the deployer in the splunk_app_for_dbconnect app.

[shclustering]
 conf_replication_include.identities = true
 conf_replication_include.db_connections = true

 This will make sure that any new identity you create from one SHC member will get replicated to other members.

You can also copy the encrypted identity from that one SH where it exists and deploy it to all SH using SHC deployer.

0 Karma

dstuder
Communicator

Those are in the default/server.conf file and they do sync across the search heads already. The issue is that the identity encrypted password is only readable on the search head that it was created on through the UI.

0 Karma

sushi
Explorer

Did you use deployer for installing db connect on search head clusters?
If you didn't use, you may need to configure with deployer.

First, install db connect on deployer and configure apps.
Next, copy db connect dir to $SPLUNK_HOME/etc/shcluster/apps/ dir.
Finally, deploy the configuration bundle by running the splunk apply and rolling restart shc.

For details, please read splunk document.

https://docs.splunk.com/Documentation/DBX/3.6.0/DeployDBX/Distributeddeployment#Deploy_DB_Connect_on... 

0 Karma

dstuder
Communicator

I did deploy it to the search heads from the search head deployer. However, the SQL identity was created through the UI and then it synced across to the other search heads. So, the SQL connections and the identities do not exist on the search head deployer.

0 Karma

sushi
Explorer

Have you configured db apps identities on deployer before deploy to shc?

0 Karma

dstuder
Communicator

Also, I don't think you can configure the identities at the command line because the password attribute is an encrypted value. I tried putting in plain text and restarting Splunk and it did not encrypt the value on it's own, so I think it has to be done through the UI. I wonder if I need to set ...

conf_replication_include.identities = false

in local/server.conf on the search head deployer, deploy it, and then create the identity on each search head manually.

https://docs.splunk.com/Documentation/DBX/3.7.0/DeployDBX/javaspec

 

0 Karma

dstuder
Communicator

No, I created the identity through the UI. So, it was synced across the search heads. It didn't get deployed from the search head deployer.

0 Karma

sushi
Explorer

As document, you should configure following steps.

1. Install db connect apps on deployer.

2. Set up identities and connections for your database on deployer from web UI.

3. Copy the splunk_app_db_connect directory from $SPLUNK_HOME/etc/apps/ to the $SPLUNK_HOME/etc/shcluster/apps/ directory on the deployer. 

4. Deploy the configuration bundle by running the splunk apply shcluster-bundle command on the deployer.

 

Note that kerberos_client.conf, identity.dat cannot be replicated to other SHC nodes after it's been modified. You need to copy the files manually to other SHC nodes.

0 Karma

dstuder
Communicator

This indeed did fix the issue. Thanks for the help.

0 Karma

dstuder
Communicator

Oh, I missed that in the docs. Thanks. I'll give that a go. I gotta say Splunk really should make a way to manage all these distributed configs in an easier fashion. It's so backwards.

0 Karma

493669
Super Champion

@dstuder Not sure on search head cluster but I would prefer to use db-connect on heavy forwarder where there won't be any password sync issue. If you can move db-connect app to Heavy forwarder would solve this issue

0 Karma

dstuder
Communicator

We do have it on the heavy forwarder for indexed data, but on the search heads this would be for search time things such as alerts and reports so alas I do still need it on the search heads as well. My users don't log into anything but the search heads.

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...