Knowledge Management

How to troubleshoot an indexer not rejoining cluster after OS rebuild and data restore of /opt/splunk/ and /var/opt/splunk?

M2016G0216
Explorer

We recently had an issue with one of our indexers. We had to do a restore of /opt/splunk and /var/opt/splunk after rebuilding the OS. When I started the splunkd service, it asked me to accept the license which I thought was strange considering this was a restore of a system that's been in production since 2015. I accepted the license and it proceeded with "upgrading" the config files. After that, the system wasn't recognized by the master node and nor could I get the indexer to rejoin the cluster. I noticed that splunkd failed to run. I re-entered the passkey in clear text for pass4SymmKey in /opt/splunk/etc/system/local/server.conf and attempted to start splunkd again. This time splunkd was able to run, but the indexer couldn't communicate on port 8000 even though in the checking prerequisites it listed port 8000 as open. I got the message "Waiting for web server at https://127.0.0.1:8000 to be available." Also, I got the following error as splunkd was attempting to start when checking conf files for problems -- Can't read key file /opt/splunk/etc/auth/server.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt.
Couldn't initialize SSL Context for HTTPClient in ServerConfig. Any recommendations on what I should do next to get the indexer to rejoin the cluster?

Tags (1)
0 Karma
1 Solution

M2016G0216
Explorer

The issue was identified and resolved -- server.pem was bad due erroneous replacment, sslkeys were reset and correct server.pem used. There remained some issues with duplicate bucket ids which had to be fixed before the indexer was able to rejoin

View solution in original post

0 Karma

M2016G0216
Explorer

The issue was identified and resolved -- server.pem was bad due erroneous replacment, sslkeys were reset and correct server.pem used. There remained some issues with duplicate bucket ids which had to be fixed before the indexer was able to rejoin

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Using the Splunk Threat Research Team’s Latest Security Content

REGISTER HERE Tech Talk | Security Edition Did you know the Splunk Threat Research Team regularly releases ...

SplunkTrust | 2024 SplunkTrust Application Period is Open!

It's that time again, folks! That's right, the application/nomination period for the 2024 SplunkTrust is ...