SAML Assertion signature verification failed. Unable to get local issuer certificate

[umesh_waghode]

We are trying to configure SAML integration for our Splunk On-Premise instance with our identity provider. Per the document, when we upload IDP Metadata, Splunk automatically creates "idpCert.pem" in /etc/auth/idpCerts/ directory.

The certificate in the "idpCert.pem" is the signing certificate that we configured at our IdP. To check that certificate is saved properly, we make a copy of it some other location ourside Splunk install directory, rename it as "idpCert.crt" and open it. Windows shows the details of the certificate.

Rest of the configuration for SAML is all fine. We are able to authenticate successfully if we disable SAML Signature Verification in authenticate.conf. But when we enable signature verification it fails with the message "Verification of SAML assertion failed".It lists "idpCert.pem" in the path.

Alternative solution discovered through self debugging and trial & error:
Modify the "idpCert.pem" to save CA certificate of the signing certificate. With this, saml assertion signature verification passes.

Question:
1. If idpCert.pem is to contain the signing certificate that Splunk parses from IdP Metadata XML, then why is signature verification failing
2. Signature verification fails even when "idpCert.pem" is modified to contain certificate chain. I created this using below command:
cat signingcert.pem signingcertCA.pem > idpCert.pem and copied this file to /etc/auth/idpCerts . But it still fails with the same message
3. Does Splunk need IdP signing cert or CA cert or both? If both, is our certificate chain creation process wrong? If it needs CA certificate only, then why is Splunk creating idpCert.pem with the signing certificate from the metadata?

Regards,
Umesh

[scannon4]

The resolution to this is in the following answers post:

https://answers.splunk.com/answers/543221/problem-with-saml-cert-error-uisaml-verification-o.html

After you add the IdP Certificate Chains (root, intermediate, leaf), go back to Authentication Method page and just click Reload Authentication at the bottom of the page. Doing this is faster that a splunk restart.

[shashanksdixit]

Hi rdimri

I am working with Umesh, who originally asked the question. I am replying on his behalf.

Thank you for your answers for both the platforms. The steps which you have given works and our issue is resolved. We have tried it for linux as well as windows. Thanks a lot.

We also would like to ask you that when it will be formally documented?
Also, currently what do you suggest to us to tell it to our customers. Do we tell them to open support ticket with you?

We would appreciate if you can reply on these questions too.

Regards,
Shashank

[rdimri_splunk]

Well we have plans to add support for this in future releases and henceforth would recommend that approach going forward. I believe in the meantime people can be pointed to this post until we have this workaround documented officially.

[shashanksdixit]

Thanks rdimri.

[rdimri_splunk]

With respect to windows the steps remain the same. However the commands to achieve what is being done are slightly different.

Navigate to the directory where splunk is installed
PS C:\Program Files\Splunk\bin>

---> here PS is the Powershell prompt <----
1)
PS C:\Program Files\Splunk\bin> .\splunk.exe cmd openssl version -d
OPENSSLDIR: "C:\wrangler-2.0\build-home\galaxy/ssl"

This gives the directory which we need to store certs under.

2) If directory above is not created then lets create it.
dir = OPENSSLDIR
if (!dir.exist()) { mkdir dir}

3) Trusted certs are stored within the certs folder of the openssl directory being used. Again lets create the certs folder if it does not exist.
dir = OPENSSLDIR/certs
if (!dir.exist()) { mkdir dir}

3.5) Find out the hashes of the certs this can be found by.
a)PS C:\Program Files\Splunk\bin> .\openssl.exe x509 -hash -noout -in 'C:\Users\Administrator\Desktop\root.pem.txt'
3221ee22
b)PS C:\Program Files\Splunk\bin> .\openssl.exe x509 -hash -noout -in 'C:\Users\Administrator\Desktop\intermediate.pem.txt'
93dafc43
c)PS C:\Program Files\Splunk\bin> .\openssl.exe x509 -hash -noout -in 'C:\Users\Administrator\Desktop\leaf.pem.txt'
44e5e200

You may see warnings like
WARNING: can't open config file: C:\wrangler-2.0\build-home\galaxy/ssl/openssl.cnf
Which are safe to ignore.

4) Create soft links to the certfiles with their hashes as their name followed by appending '.0'

PS C:\wrangler-2.0\build-home\galaxy\ssl\certs> cmd /c mklink '3221ee22.0' 'C:\Users\Administrator\Desktop\root.pem.txt'
symbolic link created for 3221ee22.0 <<===>> C:\Users\Administrator\Desktop\root.pem.txt

PS C:\wrangler-2.0\build-home\galaxy\ssl\certs> cmd /c mklink '93dafc43.0' 'C:\Users\Administrator\Desktop\intermdiate.pem.txt'
symbolic link created for 93dafc43.0 <<===>> C:\Users\Administrator\Desktop\intermdiate.pem.txt

PS C:\wrangler-2.0\build-home\galaxy\ssl\certs> cmd /c mklink '44e5e200.0' 'C:\Users\Administrator\Desktop\leaf.pem.txt'
symbolic link created for 44e5e200.0 <<===>> C:\Users\Administrator\Desktop\leaf.pem.txt

After this your directory should look something like this.

PS C:\wrangler-2.0\build-home\galaxy\ssl\certs> ls
Directory: C:\wrangler-2.0\build-home\galaxy\ssl\certs

Mode LastWriteTime Length Name
-a--- 7/11/2016 3:47 PM 0 3221ee22.0
-a--- 7/11/2016 3:49 PM 0 44e5e200.0
-a--- 7/11/2016 3:48 PM 0 93dafc43.0

You may have to do this for all intermediate certs and root cert.
as a result of 4: You should have in certs directory softlinks with names such as '1c235147.0' , 'facacbc6.0’ and so on(hash values themselves might be different).

5) Restart splunk and test if it works.

To make it clear Splunk does not use certificate store provided by microsoft, it uses one created by openssl therefore if you import certs via certlm they might not work.

[bedgar_oneok]

I can confirm that rdimri's steps solve the SAML signature verification errors.

I can also confirm that having to perform these steps feels extremely dirty. I expect better from Splunk...

[rdimri_splunk]

The xml is escaped you can unescape it to understand it better

[shashanksdixit]

Hi Dave

To add to the above answer
The attribute name (i.e. Attribute Name="role" ) in the saml response should be exactly matched with the name given in the Splunk.

Hope this helps.

Also, If you need more information, can I request you to open a new question. Because the question posted here is for a different subject. I believe if you create a new question, you will get a better response.

Regards,
Shashank

[Dave98]

We have worked through the certificate problems, but still receive the error "No valid Splunk role is found in the local mapping or in the assertion."

We have tried to follow the troubleshooting guide and have tried two different formats for the Splunk group mapping.
1) Plain Mapping -- saml_users to user splunk role.
2) DN style: "cn=keyscan,ou=dynamicgroup,ou=groups,dc=companydc"

Neither of these styles seem to work and we have checked to make sure that the mapping has a trailing semicolon and is properly formatted in the authentication.conf.

Any ideas?

[rdimri_splunk]

Dave,
Splunk needs three critical pieces of information in SAML authentication response to work well with SAML.
1) realName : This is the user name that gets populated in the UI.
2) mail: the email of the user.
3) role: this information is used to map user group names to splunk roles, an example would be "developer" -> "user" (SplunkRole), "domain admin"-> "admin" (SplunkRole). We create mappings for incoming group information to splunk roles via the SAML Group map(Green button next to SAML configuration on the UI)

Attributes in SAML Assertion should look like following.

    <saml2:Attribute Name="mail" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified">
        <saml2:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">XXXX@splunk.com</saml2:AttributeValue>
    </saml2:Attribute>
    <saml2:Attribute Name="realName" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified"><saml2:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">XXXX@splunk.com</saml2:AttributeValue>
    </saml2:Attribute>
    <saml2:Attribute Name="role" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic">
        <saml2:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">Super Admin</saml2:AttributeValue>
        <saml2:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">Everyone</saml2:AttributeValue>
    </saml2:Attribute>

You can look at what is being returned in the assertion by using a browser plugin called SAML Tracer.

[rdimri_splunk]

Hello Umesh,
Your observations are valid ones. To give you some background, historically we supported signature validation assuming that the certificate used for singing is a trusted one, that is ability to traverse up a cert chain was missing, for example if you have a cert chain of root->intermediate->leaf and leaf is the one used for signing the assertion, then we would be unable to verify it. However if there was just one trusted cert, for examples sake "certA", which could be publicly verified by itself, and assertion was signed with it it would work. We have plans in place to remedy this in forth coming releases.

If your splunk is running on *ix or Mac or Linux,you can work around this by trusting the signing certificate on the host that it is running.
I have enumerated the steps below.
Basically we want to add certs as trusted certs.
Figure out what is the openssl directory being used by splunk. There could be two openssls one that we want to use and ship with splunk and another one that is used by os. We want to add the trusted certs to the version that we use with splunk.
1) ./bin/splunk cmd openssl version -d
this should display openssl version being used by splunk.
for example:
OPENSSLDIR: "/home/build/build-home/current-BETA/openssl"
again the above value may be different based on the PATH variable set it could also point to $SPLUNK_HOME/openssl

2) If directory above is not created then lets create it.
dir = OPENSSLDIR
if (!dir.exist()) { mkdir dir}

3) Trusted certs are stored within the certs folder of the openssl directory being used. Again lets create the certs folder if it does not exist.
dir = OPENSSLDIR/certs
if (!dir.exist()) { mkdir dir}

4) Create soft links to the certfiles with their hashes as their name followed by appending '.0'
ex:
ln -s /opt/splunk/etc/auth/idpCertInter.pem `openssl x509 -hash -noout -in /opt/splunk/etc/auth/idpCertInter.pem`.0
You may have to do this for all intermediate certs and root cert.
as a result of 4: You should have in certs directory softlinks with names such as '1c235147.0' , 'facacbc6.0’ and so on(hash values themselves might be different).
5) Restart splunk and test if it works.

For windows based systems, we could probably do something similar to trust the cert.

Now getting back to the questions you have asked.
1) This works because the SAML response itself contains signing cert information, however if there is a cert chain then the parent signing cert information is not present in response. When you provide the CA cert in idpCert.pem then along with CA cert, and signing cert splunk has enough information to verify the signature.

2) I have explained (above) that it is not yet supported.
3) Again my explaination above should answer this, adding a bit of a detail splunk only extracts signing certificate because from whatever metadata I have seen till now, only signing cert is present in it instead of the entire chain. Let me know if this is not in your particular case.

Again let us know how this goes for you. Hopefully this should have demystified some of the things.

[vliggio]

Why is splunk using a build directory for these certificates instead of making the OPENSSLDIR just be /opt/splunk/openssl, and have the certs in there?

[rdimri_splunk]

Because we do not know what openssl you might have (or if have it). We also want to use an openssl version that we have tested with and meets our security policies.

[vliggio]

But you're assuming that it's the openssl version provided by splunk since you're using the cmd openssl command (and it's distributed as part of the Splunk install). It would be a lot easier to build openssl within the standard splunk hierarchy and then put the certs there by default.

[rdjoraev_splunk]

In step 2 above, could you please provide more details where to create the directory?
Would this be a full path or in reference to Splunk_Home Directory?

2) If directory above is not created then lets create it.
dir = OPENSSLDIR
if (!dir.exist()) { mkdir dir}

[rdimri_splunk]

2) the paths are absolute. value of the absolute path is obtained from step1

[davidpaper]

Hi. The directory is /home/build/build-home/current-BETA/openssl to check for existence and if it doesn't, create it.

[umesh_waghode]

When I raised this query, there was a note that usually there is a response in 24 hours. It is now more than a week but I have not received any reply.

Can someone please reply?