All Apps and Add-ons

Splunk App for AWS: Why am I getting SSL error "Certificate Verify Failed" trying to configure an account?

Contributor

When I'm attempting to add an account onto the Splunk App for AWS, I receive a SSL Certificate Verify Failed error when saving the credentials. I'm not sure how to proceed with configuration when getting this error, my Splunk environment is:

OS: openSUSE 42.1 Leap (all updates installed)
Splunk: Splunk Enterprise 6.4.2
App: Splunk App for AWS 4.2.1
Add-on: Splunk Addon for AWS 4.0.0

Full error:

08-01-2016 16:18:08.096 -0400 ERROR AdminManagerExternal - Stack trace from python handler:\nTraceback (most recent call last):\n File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 129, in init\n hand.execute(info)\n File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 589, in execute\n if self.requestedAction == ACTION_CREATE: self.handleCreate(confInfo)\n File "/opt/splunk/etc/apps/splunk_app_aws/bin/aws_accounts_handler.py", line 97, in handleCreate\n return self.handleEdit(confInfo)\n File "/opt/splunk/etc/apps/splunk_app_aws/bin/aws_accounts_handler.py", line 91, in handleEdit\n am.add_or_update(fname, keyId, secretKey, category)\n File "/opt/splunk/etc/apps/splunk_app_aws/bin/aws/aws_account_manager.py", line 120, in add_or_update\n accessible_regions = aws_utils.get_accessible_regions(self._proxy, key_id, secret_key, category)\n File "/opt/splunk/etc/apps/splunk_app_aws/bin/aws/aws_utils.py", line 605, in get_accessible_regions\n available_regions += check_commercial_regions_access(proxy, aws_access_key_id, aws_secret_access_key, token)\n File "/opt/splunk/etc/apps/splunk_app_aws/bin/aws/aws_utils.py", line 594, in check_commercial_regions_access\n return conn.get_all_regions()\n File "/opt/splunk/etc/apps/splunk_app_aws/bin/boto/ec2/connection.py", line 3493, in get_all_regions\n [('item', RegionInfo)], verb='POST')\n File "/opt/splunk/etc/apps/splunk_app_aws/bin/boto/connection.py", line 1170, in get_list\n response = self.make_request(action, params, path, verb)\n File "/opt/splunk/etc/apps/splunk_app_aws/bin/boto/connection.py", line 1116, in make_request\n return self._mexe(http_request)\n File "/opt/splunk/etc/apps/splunk_app_aws/bin/boto/connection.py", line 1030, in _mexe\n raise ex\nSSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:603)\n

I think the issue is with the OpenSSL package on this systems (and with compat with the Splunk built in version):
alt text

Here's the output of the version information:
alt text

1 Solution

Contributor

This ended up being the side effect of having a system proxy variable set in Linux. The proxy would malform the actions of the App & Addon since they take liberal usage of the REST endpoints. I removed the proxy from my config by unsetting the variable in my bashrc file and it solved the issues we were having.

View solution in original post

0 Karma

Contributor

This ended up being the side effect of having a system proxy variable set in Linux. The proxy would malform the actions of the App & Addon since they take liberal usage of the REST endpoints. I removed the proxy from my config by unsetting the variable in my bashrc file and it solved the issues we were having.

View solution in original post

0 Karma

Contributor

Well I've gotten it working, however I'm still unsure what the exact fix for it was.

I did the following and got the App & Addon working correctly:
1. Setup new VM with Ubuntu 16.04.1
2. Installed Splunk 6.4.2 (using the zipped folder, not the deb file)
3. Installed the App for AWS & Addon for AWS via the web gui (before I had been unzipping their downloads directly to /etc/apps)

I'm assuming the fix had something to do with the distro change (having a more updated version of a critical package or something). But I'm going to go back through my processes on both openSUSE and Ubuntu and try and remove all the other variables as much as I can.

0 Karma

Contributor

I've added some info to the OP with the import of the 'ssl' package in Python and the OpenSSL versions being used by Splunk; compared between Ubuntu (where the app is running fine) and OpenSUSE (where I have been getting errors configuring it).

0 Karma

Contributor

So I've narrowed it down to the distro change as the confirmed difference maker, I'm not sure what the deal is since I've got both distros set to the most recent updates. It must be something that openSUSE failed to include or is out of date.

0 Karma

Contributor

Throwing the additional error output in here as well:

08-02-2016 11:23:34.365 -0400 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/bin/runScript.py execute':    File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/boto/connection.py", line 1030, in _mexe

08-02-2016 11:23:34.365 -0400 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/bin/runScript.py execute':    File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/boto/connection.py", line 1071, in make_request

08-02-2016 11:23:34.365 -0400 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/bin/runScript.py execute':    File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/boto/s3/connection.py", line 675, in make_request

11:23:34.365 AM 
08-02-2016 11:23:34.365 -0400 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/bin/runScript.py execute':    File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/boto/s3/connection.py", line 438, in get_all_buckets

08-02-2016 11:23:34.365 -0400 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/bin/runScript.py execute':    File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws_s3buckets_handler.py", line 43, in all_buckets

08-02-2016 11:23:34.365 -0400 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python /opt/splunk/bin/runScript.py execute':    File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws_s3buckets_handler.py", line 34, in run
0 Karma

Contributor

Getting the same Errno 101 errors when attempting to do most of the configuration in the app. I do not believe its a firewall issue as this has appeared at multiple locations and on a DMZ:

Traceback (most recent call last): File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/boto/utils.py", line 210, in retry_url r = opener.open(req, timeout=timeout) File "/opt/splunk/lib/python2.7/urllib2.py", line 431, in open response = self._open(req, data) File "/opt/splunk/lib/python2.7/urllib2.py", line 449, in _open '_open', req) File "/opt/splunk/lib/python2.7/urllib2.py", line 409, in _call_chain result = func(*args) File "/opt/splunk/lib/python2.7/urllib2.py", line 1227, in http_open return self.do_open(httplib.HTTPConnection, req) File "/opt/splunk/lib/python2.7/urllib2.py", line 1197, in do_open raise URLError(err) URLError: <urlopen error [Errno 101] Network is unreachable>
0 Karma

Splunk Employee
Splunk Employee

When configure account, it checks the AK/SK in AWS. You need to have either "ec2:DescribeRegions" or "s3:ListAllMyBuckets" permission assigned. About the SSL issue, can you communicate to AWS in the Splunk instance? You can check in console of your Splunk instance directly

0 Karma

Contributor

I can confirm we have the complete set of permissions from the guide (We used the all in one premade json); and I'm able to use the awscli commands directly on my server I'm trying to run this on.

I'm not sure where the problem lies, I'm starting to think its an error with the Python libraries as part of the app/addon which need to be updated (not to mention the SSL fixes from Python 2.7.12).

0 Karma

Contributor

As an aside, I'm also seeing a bunch of 101 errors out of the URLLib tracebacks, I'm suspecting my traffic is getting blocked potentially. Does this app query other services outside of the AWS environment?

Traceback (most recent call last): File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/boto/utils.py", line 210, in retry_url r = opener.open(req, timeout=timeout) File "/opt/splunk/lib/python2.7/urllib2.py", line 431, in open response = self._open(req, data) File "/opt/splunk/lib/python2.7/urllib2.py", line 449, in _open '_open', req) File "/opt/splunk/lib/python2.7/urllib2.py", line 409, in _call_chain result = func(*args) File "/opt/splunk/lib/python2.7/urllib2.py", line 1227, in http_open return self.do_open(httplib.HTTPConnection, req) File "/opt/splunk/lib/python2.7/urllib2.py", line 1197, in do_open raise URLError(err) URLError: <urlopen error [Errno 101] Network is unreachable>
0 Karma

Contributor

I have to believe there is an error in the Python code now, I'm seeing the following in Splunk after I try to setup an account:

08-02-2016 10:21:06.548 -0400 ERROR AdminManager - Could not setup handler 'splunk_ta_aws_settings_account_region' due to missing file 'splunk_ta_aws_settings_account_region_handler.py'.  Please ensure that it is in the bin subdirectory of the appropriate Splunk app path.

Which is correct, the closest named file to that in the latest Addon/TA I've downloaded is "splunk_ta_aws_regions_handler.py"; not to mention the above is using urllib2 while they've included the bins for urllib3 (making me think that is the intended package to use).

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!