All Apps and Add-ons

AWS Add-on unable to connect to AWS due to SSL inspection with custom Root CA

jesse_corray
Explorer

When attempting to connect to AWS from within the AWS app I am receiving [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:676)

splunkd.log states:
12-14-2017 18:14:12.091 -0500 ERROR AdminManagerExternal - Stack trace from python handler:\nTraceback (most recent call last):\n File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 93, in init_persistent\n hand.execute(info)\n File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 593, in execute\n if self.requestedAction == ACTION_CREATE: self.handleCreate(confInfo)\n File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/aws_account_rh.py", line 122, in handleCreate\n exc\nRestError: REST Error [400]: Bad Request -- [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:676)\n
12-14-2017 18:14:12.091 -0500 ERROR AdminManagerExternal - Unexpected error "" from python handler: "REST Error [400]: Bad Request -- [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:676)". See splunkd.log for more details.

I ran an openssl s_client -connect sts.amazonaws.com:443 from the console and am seeing that the certificate is coming back from our SSL inspection proxy. I believe I need to add our root and intermediate certs to the correct trusted root store.

I have tried these:
$SPLUNK_HOME$/lib/python2.7/site-packages/requests/cacert.pem
$SPLUNK_HOME$/bin/3rdparty/botocore/vendored/requests/cacert.pem
$SPLUNK_HOME$/bin/3rdparty/requests/cacert.pem
Still getting the error. I know the Azure app uses its own ca cert file, but have not set this one up behind the SSL proxy before.

Does anyone know what CA cert file is used by the AWS Add-on (Splunk_TA_AWS 4.4.0 on Splunk 7.0.0) when connecting to AWS to add accounts to the add on?

1 Solution

jesse_corray
Explorer

I found the cacerts file used by the AWS app. Both our root cert and our intermediate had to be added to /opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/botocore/vendored/requests/cacert.pem to trust our corporate issued certificates.

View solution in original post

jesse_corray
Explorer

I found the cacerts file used by the AWS app. Both our root cert and our intermediate had to be added to /opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/botocore/vendored/requests/cacert.pem to trust our corporate issued certificates.

youngsuh
Contributor

which version of the add-on are you running and Splunk version?  It's not there for AWS 6.x add-on with Splunk 9.x

0 Karma

cesaccenturefed
Path Finder

How in the hell did you figure this one out!

0 Karma

jesse_corray
Explorer

I searched for every cacerts.pem on the system and found this one within the app....

0 Karma

cesaccenturefed
Path Finder

Bummer no longer getting the other issue. but I am getting this [X509] PEM lib (_ssl.c:2997)

0 Karma

dtgranger
New Member

Had the exact same issue, and this solution worked for me as well.
THANK YOU, THANK YOU, THANK YOU!!!

Too many certs in too many places if you ask me.
I searched all over the web and mostly got Python related suggestions.
I also searched all over Splunk directories and found about 20 different locations of where certs are stored, so it's hard to know which are in use and when.

Added our intermediate and root certs to the file mentioned in the solution, one I had not tried yet, and voila!
Embarrassingly, I have spent days on this issue, so thanks again!

Pasting the file location again, since it was the magic touch:
/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/botocore/vendored/requests/cacert.pem

0 Karma

nickhills
Ultra Champion

Interesting! Our proxy was flagging it as STS, So I may well give this a try.
Be sure to accept your own answer so people searching in the future can see that you solved it!

If my comment helps, please give it a thumbs up!
0 Karma

jesse_corray
Explorer

Sorry, for clarification, it is not from within the AWS App. It's from within the AWS add-on (Splunk_TA_AWS 4.4.0)

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...