Splunk Add-on for AWS Problem: Does anyone know how to disable the SSL certificate validation in the add-on?

[badrinath_itrs]

We are using the Splunk Add-on  for AWS (Version : 5.0.3) and SPLUNK version 8.0.8 . We would like to leverage the SPLUNK Add-on  to consume data from Kinesis Stream and internally send data to SPLUNK HEC end point . 

When it is sending data to Internal HEC end point  ( port : 8088)  it is throwing error as below for the self signed certificate being used for SPLUNK HEC. 

Does anyone know how to disable the SSL certificate validation in the add-on ? 

Traceback (most recent call last):
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunktalib/event_writer.py", line 252, in write_events
data=event, http=self._http)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunktalib/rest.py", line 31, in splunkd_request
data, timeout, retry)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunktalib/rest.py", line 62, in urllib3_request
data, timeout, retry, urllib3_req)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunktalib/rest.py", line 97, in do_splunkd_request
raise e
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunktalib/rest.py", line 93, in do_splunkd_request
data, timeout)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunktalib/rest.py", line 57, in urllib3_req
preload_content=True)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/request.py", line 80, in request
method, url, fields=fields, headers=headers, **urlopen_kw
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/request.py", line 171, in request_encode_body
return self.urlopen(method, url, **extra_kw)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/poolmanager.py", line 330, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/connectionpool.py", line 760, in urlopen
**response_kw
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/connectionpool.py", line 720, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/util/retry.py", line 436, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='127.0.0.1', port=8088): Max retries exceeded with url: /services/collector (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1106)')))

[JosephHobbs]

One option would be to add your internal signer to the ca certs list python is using to validate the certificate.  It still requires you modify the AddOn, but it's at least maintaining the security of the connection...

The certifi module contains the ca certificates file and can be found at SPLUNK_HOME/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/certifi/cacert.pem.  You can toss your internal signer at the end of the file...

[thambisetty]

If your Splunk version is 9.2 and above and running on Linux. 
You could try below 
https://docs.splunk.com/Documentation/Splunk/9.2.1/Admin/Serverconf

[mstadler_splunk]

We had the same issue. Adding the Proxy Certs to this file did do the trick.

/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/certifi/cacert.pem

[badrinath_itrs]

I have not tried that option but will give a try as well. 

[thambisetty]

As of version 

7.4.1

your org cert must be appended to below path: 

$SPLUNK_HOME/etc/apps/Splunk_TA_aws/lib/certifi/cacert.pem

[JosephHobbs]

Hopefully it helps.  In my situation the AWS Add-on is running behind a proxy that essentially 'middle  mans' the connection so it can be scanned by a DLP product.  The certs are signed by our internal signer, so out of the box this Add-on will not work.  Adding our root internal signer works like a champ...

Good luck!

[badrinath_itrs · Accepted solution]

Checked with SPLUNK Support and got below response.

Splunk_TA_aws is using Python's urllib3 module (https://pypi.org/project/urllib3/) across the board (not just for Kinesis), and this module is coded to do SSL certificate verification. I'm afraid tweaking it will affect the entire TA.

As of now I have modified the code to use http when sending data to HEC to resolve the issue.

Below file has been modified. 
Splunk_TA_aws/bin/3rdparty/python3/splunktalib/hec_config.py

[Raja_splunk_88]

 Hi @badrinath_itrs ,

Can you please help me to provide  modified script  as i am having same problem in my environment.

Splunk version: 8.2.3
AWS addon : 5.2.0 

Thanks

Rajasekhar.O

[badrinath_itrs]

@Raja_splunk_88 , I would not recommend to go down this path and the changes gets lost during upgrade etc.
May be better to configure the TA not to use HEC and send to normal splunk port.

If you still want to make the changes , search for hec_config.py under the TA directory and make below changes for enableSSL flag setting to 0 from 1 . 

Sample changes below. 

 if not hec_input:
        logger.info("Create HEC data input")
        hec_settings = {
           #"enableSSL": 1, ## Does not work with self signed cert 
            "enableSSL": 0,
            "port": port,
            "output_mode": "json",
            "disabled": 0,
        }