how do you encrypt app passwords in a search head cluster

[awurster]

i have an app (in this case the JIRA modular alerts app) that needs a password to be deployed within our search head cluster bundle. it uses a credential store inside passwords.conf.

i've experimented with different methods of deploying the password securely on the deployer node, and then doing a search bundle push. it seems to work okay in terms of deploying - but it just gets pushed out in the clear and then just sits on disk. i'd also like to see an official guide or response on how to deploy it properly and encrypt it.

my "push" job basically:

1. pulls down that config
2. drops in the "secret"
3. restarts the deployer (this step added to try and force the deployer to encrypt it or something)

4. and does a bundle push / rolling restart

   splunk@search-deployer:~/etc/shcluster$ cat apps/atlassian-add-on-jira-alerts/local/passwords.conf
   [credential::jira_password:]
   password =

   splunk@search-member:~$ cat etc/apps/atlassian-add-on-jira-alerts/default/passwords.conf
   [credential::jira_password:]
   password =

[hardikJsheth]

Putting this for everyone's information.

In SHC, Splunk recommends using same value for splunk.secret in all the nodes. This ensures that password encrypted at one node can be decrypted on another node. After Splunk version >= 6.3 it does this automatically when you enable Search Head Clustering on your node.

However we don't have a setup page on SHC nodes. We need to push this applications from Search head deployer node. The search head deployer node will have different splunk.secret. In order to ensure that encryption works correctly we should have same splunk.secret in Search head deployer node as we have on SHC nodes. Also copy following items in server.conf file from search head node into search head deployer node

pass4SymmKey under general stanza
sslKeysfilePassword under sslConfig stanza
pass4SymmKey under shclustering stanza

After doing this changes, restart your Search Head deployer node. Complete the setup of the application from the Search Head Deployer UI. Splunk will generate passwords.conf file in your app /local folder. Move this applicaiton to shcluster/app folder. Then fire deploy command to push this bundle to all the search head.

$SPLUNK_HOME/bin/splunk apply shcluster-bundle -target https://10.0.1.1:8089

This will ensure that you have encrypted passwords on all the nodes.

[awurster]

looks great.
i will accept the answer if i get around to testing this myself, or if you can provide a snippet to demonstrate it works.

[duffeysplunk]

Also even though Setup does not show up on Manage Applications on the SHC, you can take the URL and still run it on SHC.

Normally the URL will be of this form:
/en-US/manager/splunk-add-on-jira-alerts/apps/local/splunk-add-on-jira-alerts/setup?action=edit

I was able to edit my config by using this URL and then the encrypted password was replicated across all members.

[hardikJsheth]

Yes it works. Thanks for sharing.

[awurster]

i guess more a question then for the developer, Siggy:

why use passwords.conf instead of app.conf as shown in other working examples in dev guide? currently in other apps (like our duo add-on https://bitbucket.org/asecurityteam/duo-security-app) we use app.conf with the password endpoint that just encrypts it on a standalone search head.

does app.conf credentials get encrypted when splunk restarts? or is it the same bug?

[jworthington_sp]

Hi there,

This is more of an app issues, I think, than a SHC issue. Passwords do not currently get hashed in app directories. So we recommend that you create a different cert (with the same CA) so as not to expose the SSL password you use elsewhere. We talk about this issue (and workaround) a bit in this topic:

http://docs.splunk.com/Documentation/Splunk/6.1.9/Security/ConfigureSplunkforwardingtousesignedcerti...

Specifically:

Warning: If you configure inputs.conf or outputs.conf in an app directory, the password is NOT encrypted and the clear-text value remains in the file. For this reason, you may prefer to create different certificates (signed by the same root CA) to use when configuring SSL in app directories.

Hope that helps.

[cscotz83]

Just seems strange Splunk best practices generally say to not set configurations in etc/system/local yet passwords in etc/apps get left in clear text...

[awurster]

I downvoted this post because this has nothing to do with inputs or outputs at all. if you can show me a working example where this encrypts the app password, and/or provide an bug ID for this, then will re-vote.