Splunk Cloud Integration with ServiceNOW- "Error occurred while getting access token using auth code"

[emccauley11]

Attempting to send events/incidents to ServiceNOW from Splunk.  We've completed all of the configuration steps on the SNOW side, and when we open up the SNOW app (inside Splunk Cloud) and try to add the ServiceNOW account we get the message:

"An error occurred while trying to authenticate.  Please try again."

These are the log entries that are showing up in TA_Snow_Error_Output.  Has anyone seen this before and/or seen a way through it?

2022-01-14 19:27:00,053 ERROR pid=27053 tid=MainThread file=splunk_ta_snow_rh_oauth.py:handleEdit:106 | Error occurred while getting access token using auth code

2022-01-14 19:19:40,670 ERROR pid=17428 tid=MainThread file=splunk_ta_snow_account_validation.py:validate:119 | Failure occurred while verifying username and password. Response code=403 (Forbidden)

[emccauley11]

In case anyone is tracking this one the latest update is we were able to complete the integration on a local instance of Splunk Enterprise installed on a desktop within our network.  Taking those exact same values and trying to connect in the Splunk Cloud space results in the 403 error.  This appears to be specific to Splunk Cloud.  

[venkatasri]

Hi @emccauley11 

Error 403 is something to be fixed on SNOW side.

[JosephHobbs]

I think the 403 is unrelated and probably related to the specific test that was run.

I have attempted to configure the ServiceNow addon from both Splunk Cloud as well as a local instance of Splunk Enterprise running on my workstation.  Using the same info (client id/secret) I can establish a connection with my local Splunk instance, but experience an error when attempting to do the same in Splunk Cloud.

Splunk Support pulled the splunk_ta_snow_main.log at my request from the search head in question and the below error occurs every time we try to complete the OAuth exchange...

2022-01-24 13:03:04,742 INFO pid=12591 tid=MainThread file=splunk_ta_snow_rh_oauth.py:getProxyDetails:121 | Proxy is not enabled
2022-01-24 13:03:04,768 ERROR pid=12591 tid=MainThread file=splunk_ta_snow_rh_oauth.py:handleEdit:106 | Error occurred while getting access token using auth code
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/Splunk_TA_snow/bin/splunk_ta_snow_rh_oauth.py", line 95, in handleEdit
    content = json.loads(content)
  File "/opt/splunk/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/opt/splunk/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/opt/splunk/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Given this same exchange works from a local Splunk Enterprise instance, it tells me that this is not an issue on the ServiceNow side...

I've increased the logging level and requested another set of logs.  Hopefully it will give a little more insight into what is going on...

[svasani_splunk]

You can try adding Splunk cloud SH IP to allow list in ServiceNow

[JosephHobbs]

Oops, I thought I had replied back to this thread with the final solution.  To your point, it was an issue with the ip allow list on the ServiceNow side.  Once we added the Splunk Cloud search head IPs to the SNOW allow list we were able to connect as expected.

Thanks!