All Apps and Add-ons

Splunk App for AWS: Why am I getting SSL error "Certificate Verify Failed" trying to configure an account?

goodsellt
Contributor

When I'm attempting to add an account onto the Splunk App for AWS, I receive a SSL Certificate Verify Failed error when saving the credentials. I'm not sure how to proceed with configuration when getting this error, my Splunk environment is:

OS: openSUSE 42.1 Leap (all updates installed)
Splunk: Splunk Enterprise 6.4.2
App: Splunk App for AWS 4.2.1
Add-on: Splunk Addon for AWS 4.0.0

Full error:

08-01-2016 16:18:08.096 -0400 ERROR AdminManagerExternal - Stack trace from python handler:\nTraceback (most recent call last):\n File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 129, in init\n hand.execute(info)\n File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 589, in execute\n if self.requestedAction == ACTION_CREATE: self.handleCreate(confInfo)\n File "/opt/splunk/etc/apps/splunk_app_aws/bin/aws_accounts_handler.py", line 97, in handleCreate\n return self.handleEdit(confInfo)\n File "/opt/splunk/etc/apps/splunk_app_aws/bin/aws_accounts_handler.py", line 91, in handleEdit\n am.add_or_update(fname, keyId, secretKey, category)\n File "/opt/splunk/etc/apps/splunk_app_aws/bin/aws/aws_account_manager.py", line 120, in add_or_update\n accessible_regions = aws_utils.get_accessible_regions(self._proxy, key_id, secret_key, category)\n File "/opt/splunk/etc/apps/splunk_app_aws/bin/aws/aws_utils.py", line 605, in get_accessible_regions\n available_regions += check_commercial_regions_access(proxy, aws_access_key_id, aws_secret_access_key, token)\n File "/opt/splunk/etc/apps/splunk_app_aws/bin/aws/aws_utils.py", line 594, in check_commercial_regions_access\n return conn.get_all_regions()\n File "/opt/splunk/etc/apps/splunk_app_aws/bin/boto/ec2/connection.py", line 3493, in get_all_regions\n [('item', RegionInfo)], verb='POST')\n File "/opt/splunk/etc/apps/splunk_app_aws/bin/boto/connection.py", line 1170, in get_list\n response = self.make_request(action, params, path, verb)\n File "/opt/splunk/etc/apps/splunk_app_aws/bin/boto/connection.py", line 1116, in make_request\n return self._mexe(http_request)\n File "/opt/splunk/etc/apps/splunk_app_aws/bin/boto/connection.py", line 1030, in _mexe\n raise ex\nSSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:603)\n

I think the issue is with the OpenSSL package on this systems (and with compat with the Splunk built in version):
alt text

Here's the output of the version information:
alt text

1 Solution

goodsellt
Contributor

This ended up being the side effect of having a system proxy variable set in Linux. The proxy would malform the actions of the App & Addon since they take liberal usage of the REST endpoints. I removed the proxy from my config by unsetting the variable in my bashrc file and it solved the issues we were having.

View solution in original post

0 Karma

goodsellt
Contributor

This ended up being the side effect of having a system proxy variable set in Linux. The proxy would malform the actions of the App & Addon since they take liberal usage of the REST endpoints. I removed the proxy from my config by unsetting the variable in my bashrc file and it solved the issues we were having.

0 Karma

goodsellt
Contributor

Well I've gotten it working, however I'm still unsure what the exact fix for it was.

I did the following and got the App & Addon working correctly:
1. Setup new VM with Ubuntu 16.04.1
2. Installed Splunk 6.4.2 (using the zipped folder, not the deb file)
3. Installed the App for AWS & Addon for AWS via the web gui (before I had been unzipping their downloads directly to /etc/apps)

I'm assuming the fix had something to do with the distro change (having a more updated version of a critical package or something). But I'm going to go back through my processes on both openSUSE and Ubuntu and try and remove all the other variables as much as I can.

0 Karma

goodsellt
Contributor

I've added some info to the OP with the import of the 'ssl' package in Python and the OpenSSL versions being used by Splunk; compared between Ubuntu (where the app is running fine) and OpenSUSE (where I have been getting errors configuring it).

0 Karma

goodsellt
Contributor

So I've narrowed it down to the distro change as the confirmed difference maker, I'm not sure what the deal is since I've got both distros set to the most recent updates. It must be something that openSUSE failed to include or is out of date.

0 Karma

goodsellt
Contributor

Throwing the additional error output in here as well:

08-02-2016 11:23:34.365 -0400 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/bin/runScript.py execute':    File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/boto/connection.py", line 1030, in _mexe

08-02-2016 11:23:34.365 -0400 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/bin/runScript.py execute':    File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/boto/connection.py", line 1071, in make_request

08-02-2016 11:23:34.365 -0400 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/bin/runScript.py execute':    File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/boto/s3/connection.py", line 675, in make_request

11:23:34.365 AM 
08-02-2016 11:23:34.365 -0400 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/bin/runScript.py execute':    File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/boto/s3/connection.py", line 438, in get_all_buckets

08-02-2016 11:23:34.365 -0400 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/bin/runScript.py execute':    File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws_s3buckets_handler.py", line 43, in all_buckets

08-02-2016 11:23:34.365 -0400 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/bin/runScript.py execute':    File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws_s3buckets_handler.py", line 34, in run
0 Karma

goodsellt
Contributor

Getting the same Errno 101 errors when attempting to do most of the configuration in the app. I do not believe its a firewall issue as this has appeared at multiple locations and on a DMZ:

Traceback (most recent call last): File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/boto/utils.py", line 210, in retry_url r = opener.open(req, timeout=timeout) File "/opt/splunk/lib/python2.7/urllib2.py", line 431, in open response = self._open(req, data) File "/opt/splunk/lib/python2.7/urllib2.py", line 449, in _open '_open', req) File "/opt/splunk/lib/python2.7/urllib2.py", line 409, in _call_chain result = func(*args) File "/opt/splunk/lib/python2.7/urllib2.py", line 1227, in http_open return self.do_open(httplib.HTTPConnection, req) File "/opt/splunk/lib/python2.7/urllib2.py", line 1197, in do_open raise URLError(err) URLError: <urlopen error [Errno 101] Network is unreachable>
0 Karma

pchen_splunk
Splunk Employee
Splunk Employee

When configure account, it checks the AK/SK in AWS. You need to have either "ec2:DescribeRegions" or "s3:ListAllMyBuckets" permission assigned. About the SSL issue, can you communicate to AWS in the Splunk instance? You can check in console of your Splunk instance directly

0 Karma

goodsellt
Contributor

I can confirm we have the complete set of permissions from the guide (We used the all in one premade json); and I'm able to use the awscli commands directly on my server I'm trying to run this on.

I'm not sure where the problem lies, I'm starting to think its an error with the Python libraries as part of the app/addon which need to be updated (not to mention the SSL fixes from Python 2.7.12).

0 Karma

goodsellt
Contributor

As an aside, I'm also seeing a bunch of 101 errors out of the URLLib tracebacks, I'm suspecting my traffic is getting blocked potentially. Does this app query other services outside of the AWS environment?

Traceback (most recent call last): File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/boto/utils.py", line 210, in retry_url r = opener.open(req, timeout=timeout) File "/opt/splunk/lib/python2.7/urllib2.py", line 431, in open response = self._open(req, data) File "/opt/splunk/lib/python2.7/urllib2.py", line 449, in _open '_open', req) File "/opt/splunk/lib/python2.7/urllib2.py", line 409, in _call_chain result = func(*args) File "/opt/splunk/lib/python2.7/urllib2.py", line 1227, in http_open return self.do_open(httplib.HTTPConnection, req) File "/opt/splunk/lib/python2.7/urllib2.py", line 1197, in do_open raise URLError(err) URLError: <urlopen error [Errno 101] Network is unreachable>
0 Karma

goodsellt
Contributor

I have to believe there is an error in the Python code now, I'm seeing the following in Splunk after I try to setup an account:

08-02-2016 10:21:06.548 -0400 ERROR AdminManager - Could not setup handler 'splunk_ta_aws_settings_account_region' due to missing file 'splunk_ta_aws_settings_account_region_handler.py'.  Please ensure that it is in the bin subdirectory of the appropriate Splunk app path.

Which is correct, the closest named file to that in the latest Addon/TA I've downloaded is "splunk_ta_aws_regions_handler.py"; not to mention the above is using urllib2 while they've included the bins for urllib3 (making me think that is the intended package to use).

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...