How to configure OKTA saml authentication for a SHC Cluster without SLO errors?

[Glasses]

Scenario:

3 node SHC behind okta auth

Suppose you have a URL splunk-foo.com points to an ALB which load balances user logins between SH1, SH2, and SH3.

For example you navigate to https://splunk-foo.com > you get directed to SH1, then SH1 redirects you to an IDP (like OKTA for MFA) after you complete authentication then you are logged in.

Lets say when you initiated the OKTA -idpCert.pem  creation you used the clientcert of SH1 server.pem.  Now you will notice that when you logout from SH2 or SH3 you get an error like >

IDP failed to handle logout request.Status="Status Code="urn:oasis:names:tc:SAML:2.0:AuthnFailed"

After re-reading Splunk docs, Okta docs, Community Posts, etc (becoming thoroughly confused)… 

We inferred that OKTA needs a copy of the SH1 server.pem as the clientCert for all other SHC nodes (i.e. SH2 and SH3).  So we copied/renamed the SH1 server.pem > idp-okta.pem and dropped it in the .../etc/auth/ dir and then configured in  .../etc/system/local/ authentication.conf   the path like this>

[saml]
#clientCert = /opt/splunk/etc/auth/server.pem
clientCert = /opt/splunk/etc/auth/idp-okta.pem

Apparently this works.

However, I am wondering if this is the correct way???  

As I said before the docs are a bit cloudy regarding this OKTA setup for SHCs.   As a single search head deployment the steps would work.

Please advise if there is a better way or there is some unanticipated SSL concern with this method.

RE: >>> https://docs.splunk.com/Documentation/Splunk/8.0.6/Security/SAMLSHC

This appears to be updated recently with a new directions... or maybe we just misunderstood...

It seems that you should not submit a specific SH node server.pem to OKTA to create a idpCert, but rather create a new cert.pem and then install the new "saml" clientCert.pem and the resulting idpCert on all the SHC nodes.

As a side question, if you were to change all the SHC nodes to use the same server.pem, (i.e. replace SH2 and SH3 server.pem with SH1 server.pem) would that cause ssl to break or mess up the SHC performance?

Thank you in advance.

[codebuilder]

Use a wildcard cert and terminate SSL at the load balancer.
Also, be sure to enable sticky sessions (or the equivalent for your LB) so that the end user stays on a given search head after hitting the VIP.

[Glasses]

Hi Codebuilder,

Thank you for the reply.

Unfortunately, we require the user to authenticate to the SHC node, so terminating at the LB is not an option.

I believe the issue we had is that a "new / generic" OKTA-Splunk Cert was required, to preclude the SOL errors.

[codebuilder]

You're logging into the SHC node(s) directly instead of via a load balanced VIP?
If so, what is your use case for this?