All Apps and Add-ons

Splunk Add-on for AWS Problem with Self Signed Certificate

badrinath_itrs
Communicator

We are using the Splunk Add-on  for AWS (Version : 5.0.3) and SPLUNK version 8.0.8 . We would like to leverage the SPLUNK Add-on  to consume data from Kinesis Stream and internally send data to SPLUNK HEC end point . 

When it is sending data to Internal HEC end point  ( port : 8088)  it is throwing error as below for the self signed certificate being used for SPLUNK HEC. 

Does anyone know how to disable the SSL certificate validation in the add-on ? 

 


Traceback (most recent call last):
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunktalib/event_writer.py", line 252, in write_events
data=event, http=self._http)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunktalib/rest.py", line 31, in splunkd_request
data, timeout, retry)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunktalib/rest.py", line 62, in urllib3_request
data, timeout, retry, urllib3_req)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunktalib/rest.py", line 97, in do_splunkd_request
raise e
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunktalib/rest.py", line 93, in do_splunkd_request
data, timeout)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunktalib/rest.py", line 57, in urllib3_req
preload_content=True)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/request.py", line 80, in request
method, url, fields=fields, headers=headers, **urlopen_kw
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/request.py", line 171, in request_encode_body
return self.urlopen(method, url, **extra_kw)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/poolmanager.py", line 330, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/connectionpool.py", line 760, in urlopen
**response_kw
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/connectionpool.py", line 720, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/util/retry.py", line 436, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='127.0.0.1', port=8088): Max retries exceeded with url: /services/collector (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1106)')))

Labels (2)
0 Karma
1 Solution

badrinath_itrs
Communicator

Checked with SPLUNK Support and got below response.

Splunk_TA_aws is using Python's urllib3 module (https://pypi.org/project/urllib3/) across the board (not just for Kinesis), and this module is coded to do SSL certificate verification. I'm afraid tweaking it will affect the entire TA.

As of now I have modified the code to use http when sending data to HEC to resolve the issue.

Below file has been modified. 
Splunk_TA_aws/bin/3rdparty/python3/splunktalib/hec_config.py

View solution in original post

0 Karma

JosephHobbs
Explorer

One option would be to add your internal signer to the ca certs list python is using to validate the certificate.  It still requires you modify the AddOn, but it's at least maintaining the security of the connection...

The certifi module contains the ca certificates file and can be found at SPLUNK_HOME/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/certifi/cacert.pem.  You can toss your internal signer at the end of the file...

badrinath_itrs
Communicator

I have not tried that option but will give a try as well. 

JosephHobbs
Explorer

Hopefully it helps.  In my situation the AWS Add-on is running behind a proxy that essentially 'middle  mans' the connection so it can be scanned by a DLP product.  The certs are signed by our internal signer, so out of the box this Add-on will not work.  Adding our root internal signer works like a champ...

Good luck!

badrinath_itrs
Communicator

Checked with SPLUNK Support and got below response.

Splunk_TA_aws is using Python's urllib3 module (https://pypi.org/project/urllib3/) across the board (not just for Kinesis), and this module is coded to do SSL certificate verification. I'm afraid tweaking it will affect the entire TA.

As of now I have modified the code to use http when sending data to HEC to resolve the issue.

Below file has been modified. 
Splunk_TA_aws/bin/3rdparty/python3/splunktalib/hec_config.py

View solution in original post

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!