Deployment Architecture

pass4symkey restarts everytime

sarit_s
Communicator

Hello
i have kubernetese environment that contains :
1 SH
1 master
3 indexers in cluster

we changed the pass4symkey in all of the components and now the status is that in the indexers the key stays as it should and in the master and SH it restored to something unknown every few hours.

what is the reason and how can i solve it ?
thanks

Labels (3)
Tags (1)
0 Karma

mattymo
Splunk Employee
Splunk Employee

Hi @sarit_s 

 

I recommend you check out the Splunk Operator project on Github as we are actively in Beta and working on centralized secrets management For Splunk in Kubernetes. 

https://github.com/splunk/splunk-operator/blob/develop/docs/PasswordManagement.md

https://github.com/splunk/splunk-operator/tree/develop/docs#getting-started-with-the-splunk-operator...

Also, can you please share any of the deployment details you are using? Are you deploying this manually or by hand?

Your issue could likely be caused by non persistent data or container restarts, but hard to tell as there is many moving parts here...

- MattyMo
Tags (1)
0 Karma

effem2
Path Finder

Hi you can check the definitive configuration with BTOOL. This is very useful to identify applied configuration.
Make sure you are having the pass4SymmKey in the right place without typo. Splunk encrypts pass4SymmKey on startup(once).

/opt/splunk/bin/splunk cmd btool --debug server list general | grep pass4SymmKey
/opt/splunk/bin/splunk cmd btool --debug server list clustering | grep pass4SymmKey

 

0 Karma

sarit_s
Communicator

Hello

thanks for your answer

i do have the pass4symkey in the right place

/opt/splunk/etc/system/local/server.conf   pass4SymmKey =

the problem is that even if im setting the key with the right credentials it is resetting every once in a while and change the value of the key

so for example i will set the key now to be helloWorld, the environment will work and in two hours from now if will change itself to kuku and the environment will not work until i will change it manually to helloWorld again  

0 Karma

effem2
Path Finder

Not 100% familiar with the Kubernetes implementation. 

But make sure the mounted volumes are only used on this cluster and not destroyed.

0 Karma

sarit_s
Communicator

since in the indexers it is working i can guess that the kubernetese configuration is ok.. what can cause password reset ?

0 Karma

effem2
Path Finder

Splunk will never set pass4SymmKey on its own. Except it is not set, which it then sets it to a default key.

Either things are happening:
- conf file gets removed/changed

- $SPLUNK_HOME/etc/auth/splunk.secret gets removed/changed

0 Karma

sarit_s
Communicator

is it possible to reset the secret and set it all from scratch ?

0 Karma

effem2
Path Finder

Yes. You can edit 

/opt/splunk/etc/system/local/server.conf 

to meet your requirements. 

0 Karma

sarit_s
Communicator

i think i was misunderstood...

 

i already changed the server.conf file but once in a while the value got reset..

is there a way to reset all the keys and secrets and deploy them from scratch ?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...