topic Re: pass4symkey restarts everytime in Deployment Architecture

pass4symkey restarts everytime

sarit_s — Tue, 09 Feb 2021 11:01:09 GMT

Hello
i have kubernetese environment that contains :
1 SH
1 master
3 indexers in cluster

we changed the pass4symkey in all of the components and now the status is that in the indexers the key stays as it should and in the master and SH it restored to something unknown every few hours.

what is the reason and how can i solve it ?
thanks

Re: pass4symkey restarts everytime

effem2 — Tue, 09 Feb 2021 13:16:08 GMT

Hi you can check the definitive configuration with BTOOL. This is very useful to identify applied configuration.
Make sure you are having the pass4SymmKey in the right place without typo. Splunk encrypts pass4SymmKey on startup(once).

/opt/splunk/bin/splunk cmd btool --debug server list general | grep pass4SymmKey /opt/splunk/bin/splunk cmd btool --debug server list clustering | grep pass4SymmKey

Re: pass4symkey restarts everytime

sarit_s — Tue, 09 Feb 2021 13:24:53 GMT

Hello

thanks for your answer

i do have the pass4symkey in the right place

/opt/splunk/etc/system/local/server.conf pass4SymmKey =

the problem is that even if im setting the key with the right credentials it is resetting every once in a while and change the value of the key

so for example i will set the key now to be helloWorld, the environment will work and in two hours from now if will change itself to kuku and the environment will not work until i will change it manually to helloWorld again

Re: pass4symkey restarts everytime

effem2 — Tue, 09 Feb 2021 13:55:36 GMT

Not 100% familiar with the Kubernetes implementation.

But make sure the mounted volumes are only used on this cluster and not destroyed.

Re: pass4symkey restarts everytime

sarit_s — Wed, 10 Feb 2021 08:04:30 GMT

since in the indexers it is working i can guess that the kubernetese configuration is ok.. what can cause password reset ?

Re: pass4symkey restarts everytime

effem2 — Wed, 10 Feb 2021 08:50:02 GMT

Splunk will never set pass4SymmKey on its own. Except it is not set, which it then sets it to a default key.

Either things are happening:
- conf file gets removed/changed

- $SPLUNK_HOME/etc/auth/splunk.secret gets removed/changed

Re: pass4symkey restarts everytime

sarit_s — Wed, 10 Feb 2021 09:27:30 GMT

is it possible to reset the secret and set it all from scratch ?

Re: pass4symkey restarts everytime

effem2 — Wed, 10 Feb 2021 10:35:55 GMT

Yes. You can edit

/opt/splunk/etc/system/local/server.conf

to meet your requirements.

Re: pass4symkey restarts everytime

sarit_s — Wed, 10 Feb 2021 11:31:13 GMT

i think i was misunderstood...

i already changed the server.conf file but once in a while the value got reset..

is there a way to reset all the keys and secrets and deploy them from scratch ?

Re: pass4symkey restarts everytime

mattymo — Sun, 28 Feb 2021 14:11:45 GMT

Hi @sarit_s

I recommend you check out the Splunk Operator project on Github as we are actively in Beta and working on centralized secrets management For Splunk in Kubernetes.

https://github.com/splunk/splunk-operator/blob/develop/docs/PasswordManagement.md

https://github.com/splunk/splunk-operator/tree/develop/docs#getting-started-with-the-splunk-operator-for-kubernetes

Also, can you please share any of the deployment details you are using? Are you deploying this manually or by hand?

Your issue could likely be caused by non persistent data or container restarts, but hard to tell as there is many moving parts here...