Indexer / Search Head clusters and app for LDAP authentication- Having trouble after update

[mattbg]

Splunk Enterprise 8.2.3.3 on Linux

In our implementation, I'm using a cluster app on our Indexer and Search Head clusters to control LDAP authentication. I have two separate apps (due to different authentication needs on each) but essentially the same basic LDAP configuration.

This has been working fine since inception but we recently had to update the password used to connect to the LDAP server.

I thought it would be a matter of simply updating the password in the 'default' authentication.conf in each of the apps and then deploying an app bundle to each cluster. I assumed that the 'local' authentication.conf, which normally gets created on each node with an encrypted version of the app password, would get updated with a new encrypted password on each of the cluster nodes as part of the bundle push.

The bundle deployments worked fine, but LDAP authentication was not working afterwards.

The 'local' authentication.conf did not get updated during the app bundle push to either cluster and the way I got it working was:

1. Manually remove the app's 'local' authentication.conf from all of the Indexer and Search Head nodes

2. Do a rolling restart of each cluster

After that, the LDAP authentication worked correctly.

Is that expected? Is there a better way of doing this? Any issues with my use of 'default' / 'local' for these purposes?

Thanks in advance for any thoughts.

[VatsalJagani]

@mattbg - Generally for the custom Apps that you deploy you have two choices:

• Put everything in the local folder.
  • So your encrypted password comes in the same local file.
  • You can re-deploy when you need to change the password.
• Put everything in the default folder
  • and your encrypted password attributed will come into the local file
  • In this case, I would say don't worry about that.
  • Generally, encrypted passwords are in the local copy of the conf file. That's how Splunk config is designed to work.

I hope this helps!!!

[mattbg]

@VatsalJagani, thanks for your reply. Both of the cases you describe don't seem to deal with the issue of the local file not being updated when a new password is deployed in a 'default' conf file. This is presumably by design - the local files should not be updated by a deployment because they contain local overrides.

Deploying 'local' files in the app would not work either - those files are not designed to be deployed and values already on the host take precedence.

So, the first deployment of a password (when no 'local' file is present) works as expected, but subsequent deployments of an updated  'default' conf with a new password do not work because the local file doesn't get updated.

There's some suggestion in the Splunk docs that it's a bad idea to deploy passwords via apps - i.e. on an index cluster, you can't even write to slave-apps, so a duplicate app will be created in $SPLUNK_HOME/etc/apps. But, given that this app is very simple (only contains LDAP authentication config) I can live with that for now.

I am starting to think that removing the 'local' authentication.conf files in the app and doing a rolling restart is the best way to automate this, but it's dependent on what else you have in those local files. In my case, I only have the password.

[mattbg]

To add - if the local files are removed before the bundle push, the rolling restart is not required (on an indexer cluster a rolling restart does happen, but it's automatic as part of the bundle push).

[VatsalJagani]

@mattbg - Ohh now I understand the challenge.

Issue:

• In the search head cluster, by default, the local configuration will be merged into default.
• When you are deploying the plain-text password, it goes to default.
• Now Splunk encrypts the default password and stored the encrypted version in the local, but the default's plain-text password will stay as it is.

My Take: I would choose to deploy this from the local directory only.

• Indexer-Cluster
  • I would deploy this password from the local directory.
• Search-Head-Cluster
  • Add the below stanza into the app.conf of this specific App.
    [shclustering]
    deployer_push_mode = full
  • Now it should deploy the default and local directory both as it is. And you should be able to push the password from the local file.
    https://docs.splunk.com/Documentation/Splunk/9.0.0/DistSearch/PropagateSHCconfigurationchanges 

• As default will always create this scenario will always create an issue with password encryption.
  • Also, you don't expect people to override these changes from UI.

Rolling restart has to do with what configuration you are pushing and nothing to do with whether it's in the local folder or default folder.

I hope this helps!!!

[mattbg]

Thanks again for your replies. Member behaviour on deployer_push_mode=full shows the following. So, while it does push the contents of local, it merges them and the existing config takes precedence, so the encrypted passwords in 'local' don't get updated. This matches what I found when I tested it in my environment this morning:

---

Copies the non-local and non-user configurations to each member's app folder and overwrites the existing contents. Merges local and user configurations with the corresponding folders on the member, such that the existing configuration on the member takes precedence.

Another suggestion I've seen is to encrypt the password outside of the app, and then include the encrypted password in the app to be distributed with the bundle. This would work for the SH cluster because the splunk.secret is the same on all nodes, but I'm not sure it would work on indexers because the secrets are different.

So, I am still working with the idea that removing the local app passwords on the SHC and Indexer cluster members at the OS level before pushing the bundles out is the way to go for this particular case. Appreciate your feedback - it is making me think 🙂

[VatsalJagani]

@mattbg 

Another suggestion I've seen is to encrypt the password outside of the app, and then include the encrypted password in the app to be distributed with the bundle. This would work for the SH cluster because the splunk.secret is the same on all nodes, but I'm not sure it would work on indexers because the secrets are different.

• Use that method on the SH cluster then.
• For the Indexer cluster, you can just push the plain text and each indexer will encrypt the password separately. I'm sure you should be able to override it back from Cluster master with a new bundle push. Just make sure to keep things in the local so you don't hold the default copy with a plain text password.

I hope this helps!!!!