ServiceNow Splunk Addon (Not Secops Addon) Issues Connecting

[NapalmYourMom]

TL;DR and I'm sorry. And desperate.

So I am basically trying to get the system and application tables from ServiceNow into Splunk and was advised to use the Splunk Add-on for ServiceNow

Note that this is not the Security Operations ServiceNow integration app.

I have followed this documentation word for word but cannot establish a connection to either of the 2 ServiceNow tenants I am testing with.

The addon implies it is a network or internet issue.

Using the application logs at index=_internal sourcetype="ta_snow" the error message indicates it is either a proxy issue or a certificate issue.  I see absolutely no requests in my proxy logs and no traffic to ServiceNow from the Splunk server I have installed the addon on (although other internet destined traffic is). 

I have attempted 

1) using the cli to create the account connection and specify to disable certificate validation

2) added the Root CA of the ServiceNow tenant to the addon as advised at the bottom of the documentation I linked above.

Still, I get the same errors.  It is almost as if the addon is not recognizing any changes I make through the cli.

Curious if anyone has successfully deployed this addon and if they had to do anything special certificate-wise, or if anyone has had issues creating accounts through the cli.

I have left some of the error below

Thanks in advance!

3-05-24 22:34:36,657 ERROR pid=73561 tid=MainThread file=splunk_ta_snow_account_validation.py:validate:154 | Unable to reach ServiceNow instance at https://derp.service-now.com. The reason for failure is=Traceback (most recent call last):
File "/opt/splunk/etc/apps/Splunk_TA_snow/lib/urllib3/util/ssl_.py", line 402, in ssl_wrap_socket
context.load_verify_locations(ca_certs, ca_cert_dir, ca_cert_data)
PermissionError: [Errno 13] Permission denied
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/splunk/etc/apps/Splunk_TA_snow/lib/urllib3/connectionpool.py", line 700, in urlopen
self._prepare_proxy(conn)
File "/opt/splunk/etc/apps/Splunk_TA_snow/lib/urllib3/connectionpool.py", line 994, in _prepare_proxy
conn.connect()
File "/opt/splunk/etc/apps/Splunk_TA_snow/lib/urllib3/connection.py", line 424, in connect
tls_in_tls=tls_in_tls,
File "/opt/splunk/etc/apps/Splunk_TA_snow/lib/urllib3/util/ssl_.py", line 404, in ssl_wrap_socket
raise SSLError(e)
urllib3.exceptions.SSLError: [Errno 13] Permission denied
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/splunk/etc/apps/Splunk_TA_snow/lib/requests/adapters.py", line 499, in send
timeout=timeout,
File "/opt/splunk/etc/apps/Splunk_TA_snow/lib/urllib3/connectionpool.py", line 786, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/opt/splunk/etc/apps/Splunk_TA_snow/lib/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='derp.service-now.com', port=443): Max retries exceeded with url: /incident.do?JSONv2&sysparm_query=sys_updated_on%3E=2000-01-01+00:00:00&sysparm_record_count=1 (Caused by SSLError(PermissionError(13, 'Permission denied')))
During handling of the above exception, another exception occurred:

[NapalmYourMom]

I found my issue.

In my case, I was doing these changes as root which elevated the permissions of anything I created/modified:

splunk_ta_snow_account.conf

splunk_ta_snow_settings.conf

The ServiceNow CA cert file in $SPLUNK_HOME/etc/apps/Splunk_TA_snow

All of these things were inaccessible to the splunk user running the addon.

Fixing permissions of these files and a couple of others fixed everything.