All Apps and Add-ons

Splunk Add-on for AWS Problem: Does anyone know how to disable the SSL certificate validation in the add-on?

badrinath_itrs
Communicator

We are using the Splunk Add-on  for AWS (Version : 5.0.3) and SPLUNK version 8.0.8 . We would like to leverage the SPLUNK Add-on  to consume data from Kinesis Stream and internally send data to SPLUNK HEC end point . 

When it is sending data to Internal HEC end point  ( port : 8088)  it is throwing error as below for the self signed certificate being used for SPLUNK HEC. 

Does anyone know how to disable the SSL certificate validation in the add-on ? 

Traceback (most recent call last):
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunktalib/event_writer.py", line 252, in write_events
data=event, http=self._http)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunktalib/rest.py", line 31, in splunkd_request
data, timeout, retry)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunktalib/rest.py", line 62, in urllib3_request
data, timeout, retry, urllib3_req)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunktalib/rest.py", line 97, in do_splunkd_request
raise e
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunktalib/rest.py", line 93, in do_splunkd_request
data, timeout)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunktalib/rest.py", line 57, in urllib3_req
preload_content=True)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/request.py", line 80, in request
method, url, fields=fields, headers=headers, **urlopen_kw
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/request.py", line 171, in request_encode_body
return self.urlopen(method, url, **extra_kw)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/poolmanager.py", line 330, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/connectionpool.py", line 760, in urlopen
**response_kw
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/connectionpool.py", line 720, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/util/retry.py", line 436, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='127.0.0.1', port=8088): Max retries exceeded with url: /services/collector (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1106)')))
Labels (2)
0 Karma
1 Solution

badrinath_itrs
Communicator

Checked with SPLUNK Support and got below response.

Splunk_TA_aws is using Python's urllib3 module (https://pypi.org/project/urllib3/) across the board (not just for Kinesis), and this module is coded to do SSL certificate verification. I'm afraid tweaking it will affect the entire TA.

As of now I have modified the code to use http when sending data to HEC to resolve the issue.

Below file has been modified. 
Splunk_TA_aws/bin/3rdparty/python3/splunktalib/hec_config.py

View solution in original post

0 Karma

JosephHobbs
Explorer

One option would be to add your internal signer to the ca certs list python is using to validate the certificate.  It still requires you modify the AddOn, but it's at least maintaining the security of the connection...

The certifi module contains the ca certificates file and can be found at SPLUNK_HOME/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/certifi/cacert.pem.  You can toss your internal signer at the end of the file...

mstadler_splunk
Splunk Employee
Splunk Employee

We had the same issue. Adding the Proxy Certs to this file did do the trick.

/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/certifi/cacert.pem

 

badrinath_itrs
Communicator

I have not tried that option but will give a try as well. 

JosephHobbs
Explorer

Hopefully it helps.  In my situation the AWS Add-on is running behind a proxy that essentially 'middle  mans' the connection so it can be scanned by a DLP product.  The certs are signed by our internal signer, so out of the box this Add-on will not work.  Adding our root internal signer works like a champ...

Good luck!

badrinath_itrs
Communicator

Checked with SPLUNK Support and got below response.

Splunk_TA_aws is using Python's urllib3 module (https://pypi.org/project/urllib3/) across the board (not just for Kinesis), and this module is coded to do SSL certificate verification. I'm afraid tweaking it will affect the entire TA.

As of now I have modified the code to use http when sending data to HEC to resolve the issue.

Below file has been modified. 
Splunk_TA_aws/bin/3rdparty/python3/splunktalib/hec_config.py

0 Karma
Get Updates on the Splunk Community!

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...

Reminder! Splunk Love Promo: $25 Visa Gift Card for Your Honest SOAR Review With ...

We recently launched our first Splunk Love Special, and it's gone phenomenally well, so we're doing it again, ...