Splunk TA for Okta stopped working with ServerNotFoundError

[tryan65]

A few weeks ago our Okta feed to Splunk stopped working. I have tried everything I can think of to fix this issue. It seems to be related to a socket error based on the built in python script that comes with the TA. See below from the ta_okta.log

I have tried removing and recreating the data inputs, changing the start time stamp, modifying the python script to increase the timeout, removing and reinstalling the TA and the host can indeed be resolved with nslookup from the heavy forwarder where this TA is installed. Also had a new API key generated and added it to the data inputs.

Any help would be greatly appreciated!

bash-4.1$ tail ta_okta.log
File "/scratch/splunk/etc/apps/Splunk_TA_okta/bin/okta_rest_client.py", line 80, in request
headers=headers)
File "/scratch/splunk/etc/apps/Splunk_TA_okta/bin/splunktalib/httplib2/init.py", line 1593, in request
(response, content) = self.request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/scratch/splunk/etc/apps/Splunk_TA_okta/bin/splunktalib/httplib2/init.py", line 1335, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/scratch/splunk/etc/apps/Splunk_TA_okta/bin/splunktalib/httplib2/init_.py", line 1263, in _conn_request
raise ServerNotFoundError("Unable to find the server at %s" % conn.host)
ServerNotFoundError: Unable to find the server at company.okta.com

This is from the referenced python script for the Okta TA.

def _conn_request(self, conn, request_uri, method, body, headers):
i = 0
seen_bad_status_line = False
while i < RETRIES:
i += 1
try:
if hasattr(conn, 'sock') and conn.sock is None:
conn.connect()
conn.request(method, request_uri, body, headers)
except socket.timeout:
raise
except socket.gaierror:
conn.close()
raise ServerNotFoundError("Unable to find the server at %s" % conn.host)

[tryan65]

It turns out this was a DNS issue. When I first saw the ServerNotFound error in the logs I tried an nslookup and ping of the Okta host and they both worked so I assumed it was some other issue. But after later trying a curl to the host (like mentioned above by hozhang) it gave an unknown host error. So in doing some more searching it turns out that it is possible for nslookup to work while curl does not for DNS resolution in certain cases. So I decided to modify my /etc/resolv.conf and add a different name server and it worked!

The end issue was one of the name servers in the list at the top was failing.

Thanks for the help!

[hozhang_splunk]

According to the log, it seems the server host is not reachable. Would you try the following command to confirm it?

curl -i  -H "Accept:application/json" -H "Content-Type:application/json" -H "Authorization:<YOUR TOKEN>" https://<YOUR Okta server host>/api/v1/users/me

Thanks!

[youngsuh]

Hozhang! Thanks for posting! this help me to isolate and issue.