All Apps and Add-ons
Highlighted

AWS Add-on unable to connect to AWS due to SSL inspection with custom Root CA

Explorer

When attempting to connect to AWS from within the AWS app I am receiving [SSL: CERTIFICATEVERIFYFAILED] certificate verify failed (_ssl.c:676)

splunkd.log states:
12-14-2017 18:14:12.091 -0500 ERROR AdminManagerExternal - Stack trace from python handler:\nTraceback (most recent call last):\n File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 93, in initpersistent\n hand.execute(info)\n File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 593, in execute\n if self.requestedAction == ACTIONCREATE: self.handleCreate(confInfo)\n File "/opt/splunk/etc/apps/SplunkTAaws/bin/awsaccountrh.py", line 122, in handleCreate\n exc\nRestError: REST Error [400]: Bad Request -- [SSL: CERTIFICATEVERIFYFAILED] certificate verify failed (ssl.c:676)\n
12-14-2017 18:14:12.091 -0500 ERROR AdminManagerExternal - Unexpected error "" from python handler: "REST Error [400]: Bad Request -- [SSL: CERTIFICATE
VERIFYFAILED] certificate verify failed (ssl.c:676)". See splunkd.log for more details.

I ran an openssl s_client -connect sts.amazonaws.com:443 from the console and am seeing that the certificate is coming back from our SSL inspection proxy. I believe I need to add our root and intermediate certs to the correct trusted root store.

I have tried these:
$SPLUNKHOME$/lib/python2.7/site-packages/requests/cacert.pem
$SPLUNK
HOME$/bin/3rdparty/botocore/vendored/requests/cacert.pem
$SPLUNK_HOME$/bin/3rdparty/requests/cacert.pem
Still getting the error. I know the Azure app uses its own ca cert file, but have not set this one up behind the SSL proxy before.

Does anyone know what CA cert file is used by the AWS Add-on (SplunkTAAWS 4.4.0 on Splunk 7.0.0) when connecting to AWS to add accounts to the add on?

Highlighted

Re: AWS Add-on unable to connect to AWS due to SSL inspection with custom Root CA

Explorer

Sorry, for clarification, it is not from within the AWS App. It's from within the AWS add-on (SplunkTAAWS 4.4.0)

0 Karma
Highlighted

Re: AWS Add-on unable to connect to AWS due to SSL inspection with custom Root CA

Explorer

I found the cacerts file used by the AWS app. Both our root cert and our intermediate had to be added to /opt/splunk/etc/apps/SplunkTAaws/bin/3rdparty/botocore/vendored/requests/cacert.pem to trust our corporate issued certificates.

View solution in original post

Highlighted

Re: AWS Add-on unable to connect to AWS due to SSL inspection with custom Root CA

Ultra Champion

Interesting! Our proxy was flagging it as STS, So I may well give this a try.
Be sure to accept your own answer so people searching in the future can see that you solved it!

0 Karma
Highlighted

Re: AWS Add-on unable to connect to AWS due to SSL inspection with custom Root CA

New Member

Had the exact same issue, and this solution worked for me as well.
THANK YOU, THANK YOU, THANK YOU!!!

Too many certs in too many places if you ask me.
I searched all over the web and mostly got Python related suggestions.
I also searched all over Splunk directories and found about 20 different locations of where certs are stored, so it's hard to know which are in use and when.

Added our intermediate and root certs to the file mentioned in the solution, one I had not tried yet, and voila!
Embarrassingly, I have spent days on this issue, so thanks again!

Pasting the file location again, since it was the magic touch:
/opt/splunk/etc/apps/SplunkTAaws/bin/3rdparty/botocore/vendored/requests/cacert.pem

0 Karma
Highlighted

Re: AWS Add-on unable to connect to AWS due to SSL inspection with custom Root CA

Path Finder

How in the hell did you figure this one out!

0 Karma
Highlighted

Re: AWS Add-on unable to connect to AWS due to SSL inspection with custom Root CA

Path Finder

Bummer no longer getting the other issue. but I am getting this [X509] PEM lib (_ssl.c:2997)

0 Karma
Highlighted

Re: AWS Add-on unable to connect to AWS due to SSL inspection with custom Root CA

Explorer

I searched for every cacerts.pem on the system and found this one within the app....

0 Karma