Knowledge Management

How to troubleshoot an indexer not rejoining cluster after OS rebuild and data restore of /opt/splunk/ and /var/opt/splunk?

M2016G0216
Explorer

We recently had an issue with one of our indexers. We had to do a restore of /opt/splunk and /var/opt/splunk after rebuilding the OS. When I started the splunkd service, it asked me to accept the license which I thought was strange considering this was a restore of a system that's been in production since 2015. I accepted the license and it proceeded with "upgrading" the config files. After that, the system wasn't recognized by the master node and nor could I get the indexer to rejoin the cluster. I noticed that splunkd failed to run. I re-entered the passkey in clear text for pass4SymmKey in /opt/splunk/etc/system/local/server.conf and attempted to start splunkd again. This time splunkd was able to run, but the indexer couldn't communicate on port 8000 even though in the checking prerequisites it listed port 8000 as open. I got the message "Waiting for web server at https://127.0.0.1:8000 to be available." Also, I got the following error as splunkd was attempting to start when checking conf files for problems -- Can't read key file /opt/splunk/etc/auth/server.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt.
Couldn't initialize SSL Context for HTTPClient in ServerConfig. Any recommendations on what I should do next to get the indexer to rejoin the cluster?

Tags (1)
0 Karma
1 Solution

M2016G0216
Explorer

The issue was identified and resolved -- server.pem was bad due erroneous replacment, sslkeys were reset and correct server.pem used. There remained some issues with duplicate bucket ids which had to be fixed before the indexer was able to rejoin

View solution in original post

0 Karma

M2016G0216
Explorer

The issue was identified and resolved -- server.pem was bad due erroneous replacment, sslkeys were reset and correct server.pem used. There remained some issues with duplicate bucket ids which had to be fixed before the indexer was able to rejoin

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...