SSL Certification Verified Fail - Self signed certificate in certificate chain

[brdr]

We are adding zscaler proxy to be used by Splunk TA o365.  Our security group is providing a Root CA 4 pem file for us to use. 

Our Splunk environment runs on RHEL and our enterprise is Splunk v8.2.1. The splunk user (configured .bashrc) has http and https proxy environment variables  set to the correct entries.   In addition, we have this variable defined:

export REQUESTS_CA_BUNDLE=$SPLUNK_HOME/etc/auth/our_pem.pem

When splunk starts up we see this error and it fails to retrieve any events from the remote site. Error is:

requests.exceptions.SSLError: HTTPSConnectionPool(host='login.microsoftonline.com', port=443): Max retries exceeded with url: /url-path-made-up/oauth2/token (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1106)')))
2022-01-13 15:36:41,891 level=INFO pid=25373 tid=MainThread logger=splunksdc.collector pos=collector.py:run:254 |  | message="Modular input exited."

Talking to our security team they are wondering where the o365 TA is looking for certificates. Any help to get us past this error? Also, reading some older Answers it appear SSL Verification is turned off by default. 

This is very important to us because we have more Splunk TAs that will need to talk to zscaler proxy. 

Thank you

[brettw]

If this only happens upon a restart of the server, you will need to ensure that the variable is declared in your startup script.

Splunk will be configured to boot-start in a best practice scenario. However, if the boot-start script does not declare that variable, it will not exist when Splunk is started by init or systemd. Your shell RC file does not execute upon boot or init/systemd starting services; only upon interactive login. Keep it there. It is important for when Splunk is restarted by a human running the shell. Or automation for that matter.

Solution: declare the variable. For init, you can simply declare it just like you already do in your shell RC in the start) f. For systemd, you will declare it like this in the [Settings] stanza:

Environment="REQUESTS_CA_BUNDLE=/full/path/to/your/internal/ca/pem/file/with/no/variables"

[tkavanagh]

@brdr @nashnexagate Hello, did any of you find a solution for this? I have the same problem

[nashnexagate]

Upon receiving the SSL certificate, I've implemented a meticulous review of its contents, segregating the relevant stanzas into distinct .pem and .key files. This refined approach ensures clarity and precision in handling SSL certificates. Furthermore, to optimize our distributed environment, I've seamlessly copied these files to both instances – the Search Head and the Heavy Forwarder.

This streamlined method aims to enhance the efficiency and consistency of our SSL certificate management across our infrastructure. If you have any questions or suggestions regarding this approach, feel free to share your insights.

[nashnexagate]

Hi, I'm currently facing the same issue

[brdr]

The solution is to add your trusted cert to splunk's system cert in $SPLUNK_HOME/etc/auth file.