Getting authentication error when adding new input for Splunk Add-on for Google Cloud Platform: SSL: CERTIFICATE_VERIFY_FAILED.

[season88481]

Hi guys,

We want to onboard some data from the Cloud Storage Bucket of our GCP platform.
When adding a new input, we have this error:

Unexpected error "" from python handler: "(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:741)'),)". See splunkd.log for more details.

I searched Splunkd log, and we have these error messages:

 ERROR Failed to execute function=handleList, error=Traceback (most recent call last):
  File "/opt/splunk/etc/apps/Splunk_TA_google-cloudplatform/bin/splunktalib/common/pattern.py", line 44, in __call__
    return func(*args, **kwargs)
  File "/opt/splunk/etc/apps/Splunk_TA_google-cloudplatform/bin/splunk_ta_gcp/resthandlers/projects.py", line 38, in handleList
    res_mgr = grm.GoogleResourceManager(logger, config)
  File "/opt/splunk/etc/apps/Splunk_TA_google-cloudplatform/bin/splunk_ta_gcp/legacy/resource_manager.py", line 51, in __init__
    self._client = gwc.create_google_client(self._config)
  File "/opt/splunk/etc/apps/Splunk_TA_google-cloudplatform/bin/splunk_ta_gcp/legacy/common.py", line 210, in create_google_client
    client = discovery.build(config["service_name"], config["version"], http=http, cache_discovery=False)
  File "/opt/splunk/etc/apps/Splunk_TA_google-cloudplatform/bin/3rdparty/oauth2client/util.py", line 137, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/opt/splunk/etc/apps/Splunk_TA_google-cloudplatform/bin/3rdparty/googleapiclient/discovery.py", line 229, in build
    requested_url, discovery_http, cache_discovery, cache)
  File "/opt/splunk/etc/apps/Splunk_TA_google-cloudplatform/bin/3rdparty/googleapiclient/discovery.py", line 276, in _retrieve_discovery_doc
    resp, content = http.request(actual_url)
  File "/opt/splunk/etc/apps/Splunk_TA_google-cloudplatform/bin/httplib2shim/google_auth.py", line 201, in request
    uri, method, body=body, headers=request_headers, **kwargs)
  File "/opt/splunk/etc/apps/Splunk_TA_google-cloudplatform/bin/httplib2_helper/httplib2_py2/httplib2/__init__.py", line 2135, in request
    cachekey,
  File "/opt/splunk/etc/apps/Splunk_TA_google-cloudplatform/bin/httplib2_helper/httplib2_py2/httplib2/__init__.py", line 1796, in _request
    conn, request_uri, method, body, headers
  File "/opt/splunk/etc/apps/Splunk_TA_google-cloudplatform/bin/httplib2shim/__init__.py", line 171, in _conn_request
    raise _map_exception(e)
SSLError: (SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:741)'),)

And this:

  ERROR AdminManagerExternal - Stack trace from python handler:\nTraceback (most recent call last):\n  File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 131, in init\n    hand.execute(info)\n  File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 595, in execute\n    if self.requestedAction == ACTION_LIST:     self.handleList(confInfo)\n  File "/opt/splunk/etc/apps/Splunk_TA_google-cloudplatform/bin/splunktalib/common/pattern.py", line 44, in __call__\n    return func(*args, **kwargs)\n  File "/opt/splunk/etc/apps/Splunk_TA_google-cloudplatform/bin/splunk_ta_gcp/resthandlers/projects.py", line 38, in handleList\n    res_mgr = grm.GoogleResourceManager(logger, config)\n  File "/opt/splunk/etc/apps/Splunk_TA_google-cloudplatform/bin/splunk_ta_gcp/legacy/resource_manager.py", line 51, in __init__\n    self._client = gwc.create_google_client(self._config)\n  File "/opt/splunk/etc/apps/Splunk_TA_google-cloudplatform/bin/splunk_ta_gcp/legacy/common.py", line 210, in create_google_client\n    client = discovery.build(config["service_name"], config["version"], http=http, cache_discovery=False)\n  File "/opt/splunk/etc/apps/Splunk_TA_google-cloudplatform/bin/3rdparty/oauth2client/util.py", line 137, in positional_wrapper\n    return wrapped(*args, **kwargs)\n  File "/opt/splunk/etc/apps/Splunk_TA_google-cloudplatform/bin/3rdparty/googleapiclient/discovery.py", line 229, in build\n    requested_url, discovery_http, cache_discovery, cache)\n  File "/opt/splunk/etc/apps/Splunk_TA_google-cloudplatform/bin/3rdparty/googleapiclient/discovery.py", line 276, in _retrieve_discovery_doc\n    resp, content = http.request(actual_url)\n  File "/opt/splunk/etc/apps/Splunk_TA_google-cloudplatform/bin/httplib2shim/google_auth.py", line 201, in request\n    uri, method, body=body, headers=request_headers, **kwargs)\n  File "/opt/splunk/etc/apps/Splunk_TA_google-cloudplatform/bin/httplib2_helper/httplib2_py2/httplib2/__init__.py", line 2135, in request\n    cachekey,\n  File "/opt/splunk/etc/apps/Splunk_TA_google-cloudplatform/bin/httplib2_helper/httplib2_py2/httplib2/__init__.py", line 1796, in _request\n    conn, request_uri, method, body, headers\n  File "/opt/splunk/etc/apps/Splunk_TA_google-cloudplatform/bin/httplib2shim/__init__.py", line 171, in _conn_request\n    raise _map_exception(e)\nSSLError: (SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:741)'),)\n

So is there a way to add our own CA cert to avoid the SSL error, or is there a way to turn-off SSL verification?

Many thanks,

S

[obelixio]

The Splunk Add-on for Google Cloud Platform is using the httplib2 library. 

What worked for us was to set the HTTPLIB2_CA_CERTS environment variable in the Splunk systemd unit file and point it to the system CA bundle (in our case /etc/ssl/ca-bundle.pem). 

Have a look at 'lib/httplib2/certs.py' to understand the logic and alternative solutions. 

[sbhatnagar88]

Have you been able to rectify it. If yes , can you suggest pls.

Thanks