Search peer SSL config check- How to resolve these errors that popped up after upgrade?

[gowthammahes]

HI,

I have a standalone server which is running on 9.0.0.1 version earlier. Now it got updated to latest version of 9.0.1.  After upgrade, the upgrade readiness app scans all the apps and it showed 2 system config failed errors in Splunk Platform Compatibility Scan.

Herewith attached the error snap.
any idea on this to resolve.

Thanks in advance

[gowthammahes]

HI,

We have contacted Splunk team and created a case for both the below config issues:

1.MongoDB TLS and DNS validation check : this config issue will be resolved as told by @tlskinneriv in the below thread.

2. Search peer SSL config check : This is not compatible with splunk versions of 9.0 and higher and it is know issue in splunk. So you can click on dismiss config button on search results.

Hope this helps !!!

[tlskinneriv]

I've got the same issue as well on 9.0.1. I was able to figure out a workaround for the Splunk Web GUI not loading (500 error) though. Turns out there is still something configured to use the certs in etc/auth/splunkweb even though the configs in web.conf and server.conf clearly point to different certificates. To get the GUI to work with "requireClientCert = True" in server.conf, I had to copy my Splunk Web certificate and key to the files in etc/auth/splunkweb. The certificate supports both client and server authentication, so I presume that's the only reason it worked. This allowed the GUI to load, but still has not fixed the issues found by the Upgrade Readiness App. Were you able to find a solution for the Mongo one?

[gowthammahes]

Hi @tlskinneriv , Getting 500 internal error for splunkweb GUI. whats the workaround to load the splunk gui please?

[tlskinneriv]

The only way I was able to resolve the 500 error was to replace the certificate files (cert.pem and privkey.pem) in the $SPLUNK_HOME/etc/auth/splunkweb with my custom cert that is trusted by Splunk.

Another behavior to note is that when I enabled client certificate requirements for Splunk in server.conf, it also required the forwarders to check in initially to the deployment server with a client certificate (port 8089), which is an extra step on install, but expected behavior.

EDIT: For this to work the certificate used must also have the Client Authentication extended key usage (EKU) attribute.

[tlskinneriv]

Upon investigating the underlying Python code for the search peer check I found that these settings need to be set in server.conf:

[sslConfig]
requireClientCert = True
verifyServerCert = True

Here is the search to check those specific settings from the GUI:

| rest /services/configs/conf-server/sslConfig | table requireClientCert, verifyServerCert

Both of these values should exist and be set to 1 for the check to pass.

Interestingly, I cannot find "verifyServerCert" referenced in any documentation. Wondering if this is a bug in the check itself and the developer actually meant "sslVerifyServerCert".

While these settings are what is required by the check, it will not make the check pass. This is due the utility library used by the Upgrade Readiness App not supporting client certificate authentication. When running with what should be passing settings, entries in logs similar to the following are found like @triptraptresko posted:

09-09-2022 08:00:06.093 +0000 WARN  SSLCommon [10232 HttpDedicatedIoThread-7] - Received fatal SSL3 alert. ssl_state='error', alert_description='handshake failure'.
09-09-2022 08:00:06.093 +0000 WARN  HttpListener [10232 HttpDedicatedIoThread-7] - Socket error from 127.0.0.1:36446 while idling: error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate

This problem exists in at least version 9.0.3 (appears to be the latest version of the app as shipped with Splunk Enterprise 9.0.1) of the app. I don't have any other data to support it existing in other versions, but presume that it exists in the versions prior to 9.0.3 as well.

For a temporary workaround, the utility can be patched to support client certificate authentication with the following modifications to the "get_connection_object" method (line 721 for me) in the etc/apps/python_upgrade_readiness_app/bin/libs_py3/pura_libs_utils/pura_utils.py file:

def get_connection_object(session_key, owner=None):
    """
    Create a new connection object for oneshot.
    :param session_key: Session key of the logged in user.

    :return: oneshot connection object.
    """
    logging.info("Creating a new connection object for oneshot.")
    try:
        args = {"token": session_key}
        if owner:
            args["owner"] = owner
        # begin fix for client cert auth
        args["key_file"] = "<path to key file>"
        args["cert_file"] = "<path to cert file>"
        # end fix for client cert auth
        service = client.connect(**args)
        return service
    except Exception as e:
        logging.exception(str(e))
        return None

Restart Splunk after the code update, and the app should be able to perform the checks successfully without the peer certificate errors.

Both the occurrence of "verifyServerCert" in the check and unsupported client certificate authentication seem like bugs to me, so I'm going to try to report them appropriately.

[tlskinneriv · Accepted solution]

Upon investigating the underlying Python code for the Mongo check, I found that these settings need to be set in server.conf regardless of what you have under [sslConfig]:

[kvstore]
serverCert = <full path to cert>
sslPassword = <ssl key password> # must be set if in FIPS mode
sslVerifyServerCert = True
sslVerifyServerName = True

sslPassword must be set regardless of if your key is encrypted or not if running in FIPS mode in accordance with the server.conf docs (https://docs.splunk.com/Documentation/Splunk/latest/Admin/Serverconf). When in FIPS mode Mongo expects to get a value regardless of if it's used or not; if it's not set, Mongo will fail to start.

The code ends up checking for the sslVerify* values from an API call to a rest endpoint. This search will show you immediately if the sslVerify* values are configured:

| rest /services/configs/conf-server/kvstore | table sslVerify*

 They should both be 1 for the check to pass.

EDIT: add caveat for FIPS mode.

[mikefg]

I'm getting these same errors when running the upgrade readiness app. Do these need to be fixed prior to upgrading to 9.0? I'm on 8.2 today.

[isoutamo]

That has told here https://docs.splunk.com/Documentation/Splunk/9.0.1/Security/EnableTLSCertHostnameValidation

Except ssl password is not mentioned on this item. If it’s really needed also in this stanza you should report that it’s missing from documents.

[tlskinneriv]

Thanks for that catch. We are running in FIPS mode, which requires it to be set according to the docs.

[triptraptresko]

Am troubleshooting the issue myself. My current suspicion is it checks for mutual TLS (mTLS) between Splunk instances, from title: Search peer SSL config check. Aka Splunk-2-Splunk mutual authentication.

I have created a certificate from lets-encrypt with multipurpose. I.E be able to be used as server and client.You can check your certificate with

openssl x509 -noout -purpose -in <certificate>

Then tried to deploy current server.conf to enable S2S authentication and encryption:

[sslConfig]
sslRootCAPath = $SPLUNK_HOME/etc/apps/<REDACTED>
serverCert = $SPLUNK_HOME/etc/apps/<REDACTED>

sslAltNameToCheck = <the lets'encrypt certificate is a wildcard certificate>

#The sslVerifyServerCert setting controls the TLS certificate requirement feature.
# value of "true", the Splunk platform instance requires that any Splunk platform instance to which it connects provides a valid TLS certificate before that connection can complete
# https://docs.splunk.com/Documentation/Splunk/9.0.1/Security/EnableTLSCertHostnameValidation
sslVerifyServerCert = true

# needed in S2S verification apparently... https://docs.splunk.com/Documentation/Splunk/9.0.1/Security/ConfigTLSCertsS2S. In this sense, client is not a forwarder... but rather another Splunk instance
#requireClientCert = true
requireClientCert = false

#sslVerifyServerName only valid in splunk version 9
#sslVerifyServerName = true

#test
cliVerifyServerName = false

# https://docs.splunk.com/Documentation/Splunk/9.0.1/Admin/Serverconf
# https://community.splunk.com/t5/Deployment-Architecture/When-requireClientCert-true-is-set-in-server-conf-unable-to-run/m-p/251588

# https://medium.com/@vikashtalanki/securing-splunk-enterprise-with-ssl-eb2fb568c90e
[httpServerListener:127.0.0.1:8090]
ssl=false

This still fails the check.

Two leads: according to docs,

  requireClientCert = true 

 is required for S2S. But when enabled, splunkweb fails and won't start with my current configuration. When checking logs, error message received is "unknown CA".

 ERROR [6318a141567ff86c0f4210] __init__:522 - Socket error communicating with splunkd (error=[SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1106)), path = /services/authentication/users/splunkadmin 

Very weird. This makes me unable to actually run the test, so don't know if this is the answer.

I have a lead to follow from here:

1. Try these configurations on 9.0.1

   If you have any other suggestions, please let me know, and I will try them  

   🙂

[triptraptresko]

tried version 9.0.1. No success, RequireClientCert = true still crashes GUI, and several issues. Without requireclientcert, upgrade readiness app still showing failed for "SSL Peer Config Check" and "MongoDB ...".

I am missing something but don't know what.

At last, tried using a self-signed certificate with X509v3 Extended Key Usage: TLS Web Server Authentication, TLS Web Client Authentication, Code Signing, E-mail Protection. Confirmed that it was a multipurpose certificate (both client and server) as well, but to no avail...

GUI Error:

ERROR    [631af018aa7ff2d0185690] __init__:591-SSLerrorcommunicatingwithsplunkd, error="[SSL:TLSV1_ALERT_UNKNOWN_CA] tlsv1alertunknownca (_ssl.c:1106)", path=/services/authentication/users/splunkadmin

GUI Error as reported from backend python script:

2022-09-09 07:49:44,679 ERROR	[631af018aa7ff2d0185690] error:335 - Traceback (most recent call last):
  File "/opt/splunk/lib/python3.7/site-packages/splunk/rest/__init__.py", line 583, in simpleRequest
    serverResponse, serverContent = h.request(uri, method, headers=headers, body=payload)
  File "/opt/splunk/lib/python3.7/site-packages/httplib2/__init__.py", line 1968, in request
    cachekey,
  File "/opt/splunk/lib/python3.7/site-packages/httplib2/__init__.py", line 1626, in _request
    conn, request_uri, method, body, headers
  File "/opt/splunk/lib/python3.7/site-packages/httplib2/__init__.py", line 1532, in _conn_request
    conn.connect()
  File "/opt/splunk/lib/python3.7/site-packages/httplib2/__init__.py", line 1313, in connect
    self.sock = self._context.wrap_socket(sock, server_hostname=self.host)
  File "/opt/splunk/lib/python3.7/ssl.py", line 428, in wrap_socket
    session=session
  File "/opt/splunk/lib/python3.7/ssl.py", line 878, in _create
    self.do_handshake()
  File "/opt/splunk/lib/python3.7/ssl.py", line 1147, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1106)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cprequest.py", line 628, in respond
    self._do_respond(path_info)
  File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cprequest.py", line 687, in _do_respond
    response.body = self.handler()
  File "/opt/splunk/lib/python3.7/site-packages/cherrypy/lib/encoding.py", line 219, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/htmlinjectiontoolfactory.py", line 75, in wrapper
    resp = handler(*args, **kwargs)
  File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/routes.py", line 383, in default
    return route.target(self, **kw)
  File "</opt/splunk/lib/python3.7/site-packages/decorator.py:decorator-gen-1208>", line 2, in render
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 40, in rundecs
    return fn(*a, **kw)
  File "</opt/splunk/lib/python3.7/site-packages/decorator.py:decorator-gen-1206>", line 2, in render
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 118, in check
    return fn(self, *a, **kw)
  File "</opt/splunk/lib/python3.7/site-packages/decorator.py:decorator-gen-1205>", line 2, in render
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 166, in validate_ip
    return fn(self, *a, **kw)
  File "</opt/splunk/lib/python3.7/site-packages/decorator.py:decorator-gen-1204>", line 2, in render
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 244, in preform_sso_check
    update_session_user(sessionKey, remote_user)
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 188, in update_session_user
    en = splunk.entity.getEntity('authentication/users', user, sessionKey=sessionKey)
  File "/opt/splunk/lib/python3.7/site-packages/splunk/entity.py", line 277, in getEntity
    serverResponse, serverContent = rest.simpleRequest(uri, getargs=kwargs, sessionKey=sessionKey, raiseAllErrors=True)
  File "/opt/splunk/lib/python3.7/site-packages/splunk/rest/__init__.py", line 592, in simpleRequest
    raise splunk.SplunkdConnectionException(msg)
splunk.SplunkdConnectionException: Splunkd daemon is not responding: ('SSL error communicating with splunkd, error="[SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1106)", path=/services/authentication/users/splunkadmin',)

Other issues:

09-09-2022 08:00:06.093 +0000 WARN  SSLCommon [10232 HttpDedicatedIoThread-7] - Received fatal SSL3 alert. ssl_state='error', alert_description='handshake failure'.
09-09-2022 08:00:06.093 +0000 WARN  HttpListener [10232 HttpDedicatedIoThread-7] - Socket error from 127.0.0.1:36446 while idling: error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate

[amt]

Having the same issue. This check is from the Platform Upgrade Readiness App so it will not be located under the Health Check Items as far as I am concerned. 

[gowthammahes]

do you have any solution to solve this error?

[amt]

I'm troubleshooting the issue. I haven't discovered the solution yet as well.

[gowthammahes]

we are also troubleshooting.  might be a ssl certificate hostname validation?

[isoutamo]

Hi

This is quite interesting. I have also 9.0.1 which haven't these two health checks. Anyhow you could check what those are by the next clicks.

Settings -> MC -> Settings -> Health Check Items.

Then open those checks and see what are in Search box. That told how this check has done. Also Suggested action can help you.

r. Ismo