All Apps and Add-ons

Splunk Add-on for AWS Problem: Does anyone know how to disable the SSL certificate validation in the add-on?

badrinath_itrs
Communicator

We are using the Splunk Add-on  for AWS (Version : 5.0.3) and SPLUNK version 8.0.8 . We would like to leverage the SPLUNK Add-on  to consume data from Kinesis Stream and internally send data to SPLUNK HEC end point . 

When it is sending data to Internal HEC end point  ( port : 8088)  it is throwing error as below for the self signed certificate being used for SPLUNK HEC. 

Does anyone know how to disable the SSL certificate validation in the add-on ? 

Traceback (most recent call last):
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunktalib/event_writer.py", line 252, in write_events
data=event, http=self._http)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunktalib/rest.py", line 31, in splunkd_request
data, timeout, retry)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunktalib/rest.py", line 62, in urllib3_request
data, timeout, retry, urllib3_req)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunktalib/rest.py", line 97, in do_splunkd_request
raise e
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunktalib/rest.py", line 93, in do_splunkd_request
data, timeout)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/splunktalib/rest.py", line 57, in urllib3_req
preload_content=True)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/request.py", line 80, in request
method, url, fields=fields, headers=headers, **urlopen_kw
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/request.py", line 171, in request_encode_body
return self.urlopen(method, url, **extra_kw)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/poolmanager.py", line 330, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/connectionpool.py", line 760, in urlopen
**response_kw
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/connectionpool.py", line 720, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/util/retry.py", line 436, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='127.0.0.1', port=8088): Max retries exceeded with url: /services/collector (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1106)')))
Labels (2)
0 Karma
1 Solution

badrinath_itrs
Communicator

Checked with SPLUNK Support and got below response.

Splunk_TA_aws is using Python's urllib3 module (https://pypi.org/project/urllib3/) across the board (not just for Kinesis), and this module is coded to do SSL certificate verification. I'm afraid tweaking it will affect the entire TA.

As of now I have modified the code to use http when sending data to HEC to resolve the issue.

Below file has been modified. 
Splunk_TA_aws/bin/3rdparty/python3/splunktalib/hec_config.py

View solution in original post

0 Karma

JosephHobbs
Path Finder

One option would be to add your internal signer to the ca certs list python is using to validate the certificate.  It still requires you modify the AddOn, but it's at least maintaining the security of the connection...

The certifi module contains the ca certificates file and can be found at SPLUNK_HOME/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/certifi/cacert.pem.  You can toss your internal signer at the end of the file...

mstadler_splunk
Splunk Employee
Splunk Employee

We had the same issue. Adding the Proxy Certs to this file did do the trick.

/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/certifi/cacert.pem

 

badrinath_itrs
Communicator

I have not tried that option but will give a try as well. 

JosephHobbs
Path Finder

Hopefully it helps.  In my situation the AWS Add-on is running behind a proxy that essentially 'middle  mans' the connection so it can be scanned by a DLP product.  The certs are signed by our internal signer, so out of the box this Add-on will not work.  Adding our root internal signer works like a champ...

Good luck!

badrinath_itrs
Communicator

Checked with SPLUNK Support and got below response.

Splunk_TA_aws is using Python's urllib3 module (https://pypi.org/project/urllib3/) across the board (not just for Kinesis), and this module is coded to do SSL certificate verification. I'm afraid tweaking it will affect the entire TA.

As of now I have modified the code to use http when sending data to HEC to resolve the issue.

Below file has been modified. 
Splunk_TA_aws/bin/3rdparty/python3/splunktalib/hec_config.py

0 Karma

Raja_splunk_88
Observer

 Hi @badrinath_itrs ,

Can you please help me to provide  modified script  as i am having same problem in my environment.

Splunk version: 8.2.3
AWS addon : 5.2.0 

Thanks

Rajasekhar.O

0 Karma

badrinath_itrs
Communicator

@Raja_splunk_88 , I would not recommend to go down this path and the changes gets lost during upgrade etc.
May be better to configure the TA not to use HEC and send to normal splunk port.

 

If you still want to make the changes , search for hec_config.py under the TA directory and make below changes for enableSSL flag setting to 0 from 1 . 

Sample changes below. 

 if not hec_input:
        logger.info("Create HEC data input")
        hec_settings = {
           #"enableSSL": 1, ## Does not work with self signed cert 
            "enableSSL": 0,
            "port": port,
            "output_mode": "json",
            "disabled": 0,
        }

 

0 Karma
Get Updates on the Splunk Community!

New Splunk Observability innovations: Deeper visibility and smarter alerting to ...

You asked, we delivered. Splunk Observability Cloud has several new innovations giving you deeper visibility ...

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...