Splunk ES not starting when indexer search was enabled from UI and then restarted.

[skiransecops]

I was recently working on Splunk Enterprise security to have a forwarder installed on the Linux machine and display it on the server. While working on this, I noticed that indexer search option is in red status. So , I went ahead and enabled the suggestion the system was asking. After that th server asked for a restart and now, it won't come up online. Could any one help here please?

below is the log when I run Splunk start

Done
[ OK ]

Waiting for web server at https://127.0.0.1:8000 to be available..............

WARNING: web interface does not seem to be available!

Further in the file: /opt/splunk/var/log/splunk/splunkd.log This is what I see - 

03-17-2024 12:10:19.240 +0000 ERROR ClusteringMgr [33823 MainThread] - pass4SymmKey setting in the clustering or general stanza of server.conf is set to empty or the default value. You must change it to a different value.
03-17-2024 12:10:19.242 +0000 ERROR loader [33823 MainThread] - clustering initialization failed; won't start splunkd

I changed the pass4symmkey and it did not help. Could any one help here please?

[skiransecops]

Hi Marnall - Enabled indexers on all.

The configuration was set on /opt/opt/splunk/etc/system/local/server.conf

[marnall]

It sounds like it is still not happy with your pass4SymmKey.

Could you say how many indexers and search heads are you using, whether this problem affects one or all search heads or indexers, and in which configuration file you've set the pass4SymmKey?