Had the very same issue. Are you using custom certificates? In that case, you might have had the same issue as me. Wrote about our the solution here: https://community.splunk.com/t5/Deployment-Architecture/KVStore-does-not-start-when-running-Splunk-9-4-WITH-A-SOLUTION/m-p/743791#M29350 Note that splunk themselves says that custom certificates are not supported, but we got it to work 😄 ... View more

TLDR: I was missing 1. outputs.conf on the sender (client) useSSL = true Thank you to michael_bates_1 , in thread https://community.splunk.com/t5/Getting-Data-In/Why-am-I-having-trouble-with-TLS/m-p/634513/highlight/false#M108573 After following the documentation on how to enable ssl between forwarders and indexers, i got the error   ERROR TcpInputProc - Error encountered for connection from src=10.1.1.34:50772. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol   In the documentation: https://docs.splunk.com/Documentation/Splunk/9.0.1/Security/ConfigureSplunkforwardingtousesignedcertificates#Configuration_file_examples_for_configuring_TLS_certificates_on_receiving_indexers , it specifies [SSL] requireClientCert = true Which if you drop, will affect your outputs.conf -> useSSL. It says if requireClientCert is defined, then useSSL will be true. In my case, I mindlessly thought you could set requrieClientCert=false... https://docs.splunk.com/Documentation/Splunk/latest/Admin/Outputsconf #----Secure Sockets Layer (SSL) Settings---- # To set up SSL on the forwarder, set the following setting/value pairs. # If you want to use SSL for authentication, add a stanza for each receiver # that must be certified. useSSL = <true|false|legacy> * Whether or not the forwarder uses SSL to connect to the receiver, or relies on the 'clientCert' setting to be active for SSL connections. * You do not need to set 'clientCert' if 'requireClientCert' is set to "false" on the receiver. * A value of "true" means the forwarder uses SSL to connect to the receiver. * A value of "false" means the forwarder does not use SSL to connect to the receiver. * The special value "legacy" means the forwarder uses the 'clientCert' property to determine whether or not to use SSL to connect. * Default: legacy   ... View more

TLDR: I was missing 1. outputs.conf on the sender (client) useSSL = true After following the documentation on how to enable ssl between forwarders and indexers, i got the error   ERROR TcpInputProc - Error encountered for connection from src=10.1.1.34:50772. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol   In the documentation: https://docs.splunk.com/Documentation/Splunk/9.0.1/Security/ConfigureSplunkforwardingtousesignedcertificates#Configuration_file_examples_for_configuring_TLS_certificates_on_receiving_indexers , it specifies [SSL] requireClientCert = true Which if you drop, will affect your outputs.conf -> useSSL. It says if requireClientCert is defined, then useSSL will be true. In my case, I mindlessly thought you could set requrieClientCert=false... https://docs.splunk.com/Documentation/Splunk/latest/Admin/Outputsconf #----Secure Sockets Layer (SSL) Settings---- # To set up SSL on the forwarder, set the following setting/value pairs. # If you want to use SSL for authentication, add a stanza for each receiver # that must be certified. useSSL = <true|false|legacy> * Whether or not the forwarder uses SSL to connect to the receiver, or relies on the 'clientCert' setting to be active for SSL connections. * You do not need to set 'clientCert' if 'requireClientCert' is set to "false" on the receiver. * A value of "true" means the forwarder uses SSL to connect to the receiver. * A value of "false" means the forwarder does not use SSL to connect to the receiver. * The special value "legacy" means the forwarder uses the 'clientCert' property to determine whether or not to use SSL to connect. * Default: legacy   ... View more

Hey! I had the same issue. If you look in   index=_internal log_level IN ("error", "warn") splunk_app_addon-builder   You'll find   /opt/splunk/etc/apps/splunk_app_addon-builder/appserver/controllers/app_validation.py cannot be executed by Python: cannot import name 'soft_unicode' from 'markupsafe' (/opt/splunk/lib/python3.7/site-packages/markupsafe/__init__.py)     Seems the python package markupsafe removed 'soft_unicode' , starting from versions later than 2.0.0. Solution is to use pip to downgrade   pip3 install --target=/opt/splunk/lib/python3.7/site-packages markupsafe==2.0.0 --force --upgrade   Then restart splunk. As I'm sure you're aware, this is absolutely a hack... Happy app-building 🙂 ... View more

I actually had to delete the registry keys, in order to install UF again. ... View more

Thank you @Richfez, worked flawlessly! ... View more

tried version 9.0.1. No success, RequireClientCert = true still crashes GUI, and several issues. Without requireclientcert, upgrade readiness app still showing failed for "SSL Peer Config Check" and "MongoDB ...". I am missing something but don't know what. At last, tried using a self-signed certificate with X509v3 Extended Key Usage: TLS Web Server Authentication, TLS Web Client Authentication, Code Signing, E-mail Protection. Confirmed that it was a multipurpose certificate (both client and server) as well, but to no avail...     GUI Error:   ERROR [631af018aa7ff2d0185690] __init__:591-SSLerrorcommunicatingwithsplunkd, error="[SSL:TLSV1_ALERT_UNKNOWN_CA] tlsv1alertunknownca (_ssl.c:1106)", path=/services/authentication/users/splunkadmin GUI Error as reported from backend python script: 2022-09-09 07:49:44,679 ERROR [631af018aa7ff2d0185690] error:335 - Traceback (most recent call last): File "/opt/splunk/lib/python3.7/site-packages/splunk/rest/__init__.py", line 583, in simpleRequest serverResponse, serverContent = h.request(uri, method, headers=headers, body=payload) File "/opt/splunk/lib/python3.7/site-packages/httplib2/__init__.py", line 1968, in request cachekey, File "/opt/splunk/lib/python3.7/site-packages/httplib2/__init__.py", line 1626, in _request conn, request_uri, method, body, headers File "/opt/splunk/lib/python3.7/site-packages/httplib2/__init__.py", line 1532, in _conn_request conn.connect() File "/opt/splunk/lib/python3.7/site-packages/httplib2/__init__.py", line 1313, in connect self.sock = self._context.wrap_socket(sock, server_hostname=self.host) File "/opt/splunk/lib/python3.7/ssl.py", line 428, in wrap_socket session=session File "/opt/splunk/lib/python3.7/ssl.py", line 878, in _create self.do_handshake() File "/opt/splunk/lib/python3.7/ssl.py", line 1147, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1106) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cprequest.py", line 628, in respond self._do_respond(path_info) File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cprequest.py", line 687, in _do_respond response.body = self.handler() File "/opt/splunk/lib/python3.7/site-packages/cherrypy/lib/encoding.py", line 219, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/htmlinjectiontoolfactory.py", line 75, in wrapper resp = handler(*args, **kwargs) File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/routes.py", line 383, in default return route.target(self, **kw) File "&lt;/opt/splunk/lib/python3.7/site-packages/decorator.py:decorator-gen-1208&gt;", line 2, in render File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 40, in rundecs return fn(*a, **kw) File "&lt;/opt/splunk/lib/python3.7/site-packages/decorator.py:decorator-gen-1206&gt;", line 2, in render File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 118, in check return fn(self, *a, **kw) File "&lt;/opt/splunk/lib/python3.7/site-packages/decorator.py:decorator-gen-1205&gt;", line 2, in render File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 166, in validate_ip return fn(self, *a, **kw) File "&lt;/opt/splunk/lib/python3.7/site-packages/decorator.py:decorator-gen-1204&gt;", line 2, in render File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 244, in preform_sso_check update_session_user(sessionKey, remote_user) File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 188, in update_session_user en = splunk.entity.getEntity('authentication/users', user, sessionKey=sessionKey) File "/opt/splunk/lib/python3.7/site-packages/splunk/entity.py", line 277, in getEntity serverResponse, serverContent = rest.simpleRequest(uri, getargs=kwargs, sessionKey=sessionKey, raiseAllErrors=True) File "/opt/splunk/lib/python3.7/site-packages/splunk/rest/__init__.py", line 592, in simpleRequest raise splunk.SplunkdConnectionException(msg) splunk.SplunkdConnectionException: Splunkd daemon is not responding: ('SSL error communicating with splunkd, error="[SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1106)", path=/services/authentication/users/splunkadmin',) Other issues: 09-09-2022 08:00:06.093 +0000 WARN SSLCommon [10232 HttpDedicatedIoThread-7] - Received fatal SSL3 alert. ssl_state='error', alert_description='handshake failure'. 09-09-2022 08:00:06.093 +0000 WARN HttpListener [10232 HttpDedicatedIoThread-7] - Socket error from 127.0.0.1:36446 while idling: error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate             ... View more

Am troubleshooting the issue myself. My current suspicion is it checks for mutual TLS (mTLS) between Splunk instances, from title: Search peer SSL config check. Aka Splunk-2-Splunk mutual authentication. I have created a certificate from lets-encrypt with multipurpose. I.E be able to be used as server and client.You can check your certificate with openssl x509 -noout -purpose -in <certificate> Then tried to deploy current server.conf to enable S2S authentication and encryption:   [sslConfig] sslRootCAPath = $SPLUNK_HOME/etc/apps/<REDACTED> serverCert = $SPLUNK_HOME/etc/apps/<REDACTED> sslAltNameToCheck = <the lets'encrypt certificate is a wildcard certificate> #The sslVerifyServerCert setting controls the TLS certificate requirement feature. # value of "true", the Splunk platform instance requires that any Splunk platform instance to which it connects provides a valid TLS certificate before that connection can complete # https://docs.splunk.com/Documentation/Splunk/9.0.1/Security/EnableTLSCertHostnameValidation sslVerifyServerCert = true # needed in S2S verification apparently... https://docs.splunk.com/Documentation/Splunk/9.0.1/Security/ConfigTLSCertsS2S. In this sense, client is not a forwarder... but rather another Splunk instance #requireClientCert = true requireClientCert = false #sslVerifyServerName only valid in splunk version 9 #sslVerifyServerName = true #test cliVerifyServerName = false # https://docs.splunk.com/Documentation/Splunk/9.0.1/Admin/Serverconf # https://community.splunk.com/t5/Deployment-Architecture/When-requireClientCert-true-is-set-in-server-conf-unable-to-run/m-p/251588 # https://medium.com/@vikashtalanki/securing-splunk-enterprise-with-ssl-eb2fb568c90e [httpServerListener:127.0.0.1:8090] ssl=false   This still fails the check. Two leads: according to docs,     requireClientCert = true     is required for S2S. But when enabled, splunkweb fails and won't start with my current configuration. When checking logs, error message received is "unknown CA".    ERROR [6318a141567ff86c0f4210] __init__:522 - Socket error communicating with splunkd (error=[SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1106)), path = /services/authentication/users/splunkadmin    Very weird. This makes me unable to actually run the test, so don't know if this is the answer. I have a lead to follow from here: Try these configurations on 9.0.1 If you have any other suggestions, please let me know, and I will try them     🙂                     ... View more