AWS Add-on unable to connect to AWS due to SSL inspection with custom Root CA

[jesse_corray]

When attempting to connect to AWS from within the AWS app I am receiving [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:676)

splunkd.log states:
12-14-2017 18:14:12.091 -0500 ERROR AdminManagerExternal - Stack trace from python handler:\nTraceback (most recent call last):\n File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 93, in init_persistent\n hand.execute(info)\n File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 593, in execute\n if self.requestedAction == ACTION_CREATE: self.handleCreate(confInfo)\n File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/aws_account_rh.py", line 122, in handleCreate\n exc\nRestError: REST Error [400]: Bad Request -- [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:676)\n
12-14-2017 18:14:12.091 -0500 ERROR AdminManagerExternal - Unexpected error "" from python handler: "REST Error [400]: Bad Request -- [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:676)". See splunkd.log for more details.

I ran an openssl s_client -connect sts.amazonaws.com:443 from the console and am seeing that the certificate is coming back from our SSL inspection proxy. I believe I need to add our root and intermediate certs to the correct trusted root store.

I have tried these:
$SPLUNK_HOME$/lib/python2.7/site-packages/requests/cacert.pem
$SPLUNK_HOME$/bin/3rdparty/botocore/vendored/requests/cacert.pem
$SPLUNK_HOME$/bin/3rdparty/requests/cacert.pem
Still getting the error. I know the Azure app uses its own ca cert file, but have not set this one up behind the SSL proxy before.

Does anyone know what CA cert file is used by the AWS Add-on (Splunk_TA_AWS 4.4.0 on Splunk 7.0.0) when connecting to AWS to add accounts to the add on?

[jesse_corray · Accepted solution]

I found the cacerts file used by the AWS app. Both our root cert and our intermediate had to be added to /opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/botocore/vendored/requests/cacert.pem to trust our corporate issued certificates.

[youngsuh]

which version of the add-on are you running and Splunk version?  It's not there for AWS 6.x add-on with Splunk 9.x

[cesaccenturefed]

How in the hell did you figure this one out!

[jesse_corray]

I searched for every cacerts.pem on the system and found this one within the app....

[cesaccenturefed]

Bummer no longer getting the other issue. but I am getting this [X509] PEM lib (_ssl.c:2997)

[dtgranger]

Had the exact same issue, and this solution worked for me as well.
THANK YOU, THANK YOU, THANK YOU!!!

Too many certs in too many places if you ask me.
I searched all over the web and mostly got Python related suggestions.
I also searched all over Splunk directories and found about 20 different locations of where certs are stored, so it's hard to know which are in use and when.

Added our intermediate and root certs to the file mentioned in the solution, one I had not tried yet, and voila!
Embarrassingly, I have spent days on this issue, so thanks again!

Pasting the file location again, since it was the magic touch:
/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/botocore/vendored/requests/cacert.pem

[nickhills]

Interesting! Our proxy was flagging it as STS, So I may well give this a try.
Be sure to accept your own answer so people searching in the future can see that you solved it!

[jesse_corray]

Sorry, for clarification, it is not from within the AWS App. It's from within the AWS add-on (Splunk_TA_AWS 4.4.0)