Getting errors messages Health Check: msg="A script exited abnormally with exit status:1" input="../configuration_chekc.py" after upgrading from Splunk Enterprise Security 5.1.1 to 6.1.0

[schandrasekar]

We have upgraded Splunk Enterprise recently to 8.0.2.1 and all the apps in our environment to the latest version. One of them is the Splunk Enterprise Security app to 6.1.0. We started receiving errors messages as "Health Check: msg="A script exited abnormally with exit status:1" input="opt/splunk/etc/apps/SA-Utils/bin/configuration_check.py stanza="configuration_check://confcheck_escorrelationmigration" .
Similar errors are popping for all the input stanzas in SplunkEnterpriseSecuritySuite configuration_check://

[sruthimadhu]

Hi, I am also facing the same issue post Splunk upgrade, is there any solution.. 

Health Check: msg="A script exited abnormally with exit status: 1" input="./opt/splunk/etc/apps/duo_splunkapp/bin/duo_input.py" stanza="duo_input://PureConnectCloud Splunk Logs"

[kkrises]

@sruthimadhu Two questions : 

1. Share your Splunk versions - previous and updated.
2. What is the version of duo_splunk app in use and the monitoring stanza content for duo_input://PureConnectCloud Splunk Logs".

The error clearly pointing out that there is an issue in executing your monitoring input stanza PureConnectCloud.

Refer this from Splunk.

Troubleshoot script errors in Splunk Enterprise Security - Splunk Documentation

[sruthimadhu]

@kkrises Splunk versions - previous 7.2.1 and updated 8.2.6

And Duo connector app previous 1.1.2 and updated 1.1.9

Stanza consists of inputs

[duo_input://PureConnectCloud Splunk Logs]
api_host = api-XXXXXXX
host = duo_XX_XXX
ikey = XXXXXXXXXX
index = duo
interval = 600
skey = XXXXXXXXXXX
sourcetype = json
source = duo

[hettervik]

In your case the error message seems to be linked to a specific script, in a specific stanza, in a specific app. I would look into that particular input script, "duo_input.py". Perhaps it's using unsupported Python 2 (as opposed to Python 3)?

[sruthimadhu]

We are using python 3.7 and multiple python versions are installed on the server, but in $SPLUNK_HOME/etc/system/local we set python.version = force_python3

[hettervik]

Hi. Did you find out why you are getting these error messages? We too are getting a lot of errors related to "authentication exception when executing configuration check" and "[HTTP 401] Client is not authenticated" from the configuration_check.py. We can't figure out why.

[DavidHourani]

Hi @schandrasekar, in Splunk 8.0 more items have been added to warning/error alerts, check to make sure you don't leave everything active as it could be overwhelming.
Also could you please have a look in your internal logs and share additional errors related to this issue ?

[schandrasekar]

Hi David , Thanks for your response . Here is the python_modular_input.log
2020-04-28 01:09:28,980+0000 ERROR pid=32226 tid=MainThread file=base_modinput.py:execute:773 | Execution failed: [HTTP 401] Client is not authenticated
Traceback (most recent call last):
File "/opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/lib/SplunkEnterpriseSecuritySuite/modinput/base_modinput.py", line 764, in execute
always_run=always_run)
File "/opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/lib/SplunkEnterpriseSecuritySuite/modinput/base_modinput.py", line 315, in do_run
self.run(stanzas)
File "/opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/bin/ess_content_importer.py", line 199, in run
exec_status, exec_status_msg = should_execute(session_key=self.input_config.session_key)
File "/opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/lib/SplunkEnterpriseSecuritySuite/pooling.py", line 186, in should_execute
if is_cluster_member(session_key):
File "/opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/lib/SplunkEnterpriseSecuritySuite/pooling.py", line 53, in is_cluster_member
r, c = splunk.rest.simpleRequest(uri, sessionKey=session_key, getargs=getargs)
File "/opt/splunk/lib/python3.7/site-packages/splunk/rest/init.py", line 553, in simpleRequest
raise splunk.AuthenticationFailed
splunk.AuthenticationFailed: [HTTP 401] Client is not authenticated
2020-04-28 01:09:29,057+0000 INFO pid=32351 tid=MainThread file=base_modinput.py:execute:718 | Execute called
2020-04-28 01:09:29,218+0000 ERROR pid=32351 tid=MainThread file=base_modinput.py:execute:773 | Execution failed: [HTTP 401] Client is not authenticated
Traceback (most recent call last):
File "/opt/splunk/etc/apps/SA-Utils/lib/SolnCommon/modinput/base_modinput.py", line 764, in execute
always_run=always_run)
File "/opt/splunk/etc/apps/SA-Utils/lib/SolnCommon/modinput/base_modinput.py", line 315, in do_run
self.run(stanzas)
File "/opt/splunk/etc/apps/DA-ESS-ThreatIntelligence/bin/threat_intelligence_manager.py", line 890, in run
self._stanza_name)
File "/opt/splunk/etc/apps/DA-ESS-ThreatIntelligence/bin/threat_intelligence_manager.py", line 690, in getStanzaNamespace
response, content = splunk.rest.simpleRequest(uri, getargs=getargs, sessionKey=session_key)
File "/opt/splunk/lib/python3.7/site-packages/splunk/rest/init_.py", line 553, in simpleRequest
raise splunk.AuthenticationFailed
splunk.AuthenticationFailed: [HTTP 401] Client is not authenticated
2020-04-28 01:10:28,836+0000 INFO pid=1547 tid=MainThread file=base_modinput.py:execute:718 | Execute called
2020-04-28 01:10:29,027+0000 INFO pid=1685 tid=MainThread file=base_modinput.py:execute:718 | Execute called
2020-04-28 01:10:29,082+0000 INFO pid=1695 tid=MainThread file=base_modinput.py:execute:718 | Execute called
2020-04-28 01:10:29,107+0000 INFO pid=1656 tid=MainThread file=base_modinput.py:execute:718 | Execute called
2020-04-28 01:10:29,126+0000 ERROR pid=1685 tid=MainThread file=base_modinput.py:execute:773 | Execution failed: [HTTP 401] Client is not authenticated
Traceback (most recent call last):
File "/opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/lib/SplunkEnterpriseSecuritySuite/modinput/base_modinput.py", line 764, in execute
always_run=always_run)
File "/opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/lib/SplunkEnterpriseSecuritySuite/modinput/base_modinput.py", line 315, in do_run
self.run(stanzas)
File "/opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/bin/app_permissions_manager.py", line 214, in run

Here is the configuration_check.log

2020-04-25 12:42:30,124+0000 INFO pid=17198 tid=MainThread file=configuration_check.py:run:135 | status="retrieved task" task="confcheck_app_exports"
2020-04-25 12:42:30,241+0000 ERROR pid=17198 tid=MainThread file=configuration_check.py:run:277 | status="Authentication exception when executing configuration check" exc="[HTTP 401] Client is not authenticated"
Traceback (most recent call last):
File "/opt/splunk/etc/apps/SA-Utils/bin/configuration_check.py", line 139, in run
entity_id, sessionKey=self.input_config.session_key)
File "/opt/splunk/lib/python3.7/site-packages/splunk/models/base.py", line 572, in get
return SplunkRESTManager(cls, sessionKey=sessionKey).get(id)
File "/opt/splunk/lib/python3.7/site-packages/splunk/models/base.py", line 552, in get
entity = self._get_entity(id, host_path=host_path)
File "/opt/splunk/lib/python3.7/site-packages/splunk/models/base.py", line 468, in _get_entity
return self._fix_entity(splunk.entity.getEntity(self.model.resource, None, sessionKey=self.sessionKey, uri=id))
File "/opt/splunk/lib/python3.7/site-packages/splunk/entity.py", line 276, in getEntity
serverResponse, serverContent = rest.simpleRequest(uri, getargs=kwargs, sessionKey=sessionKey, raiseAllErrors=True)
File "/opt/splunk/lib/python3.7/site-packages/splunk/rest/init.py", line 553, in simpleRequest
raise splunk.AuthenticationFailed
splunk.AuthenticationFailed: [HTTP 401] Client is not authenticated
2020-04-25 12:42:30,243+0000 INFO pid=17198 tid=MainThread file=configuration_check.py:run:299 | status="exiting" exit_status="2"
2020-04-25 12:43:30,108+0000 INFO pid=18332 tid=MainThread file=configuration_check.py::304 | status="starting"
2020-04-25 12:43:30,110+0000 INFO pid=18332 tid=MainThread file=configuration_check.py:run:129 | status="executing"
2020-04-25 12:43:30,110+0000 INFO pid=18332 tid=MainThread file=configuration_check.py:run:135 | status="retrieved task" task="confcheck_app_exports"
2020-04-25 12:43:30,220+0000 ERROR pid=18332 tid=MainThread file=configuration_check.py:run:277 | status="Authentication exception when executing configuration check" exc="[HTTP 401] Client is not authenticated"
Traceback (most recent call last):
File "/opt/splunk/etc/apps/SA-Utils/bin/configuration_check.py", line 139, in run
entity_id, sessionKey=self._input_config.session_key)
File "/opt/splunk/lib/python3.7/site-packages/splunk/models/base.py", line 572, in get
return SplunkRESTManager(cls, sessionKey=sessionKey).get(id)
File "/opt/splunk/lib/python3.7/site-packages/splunk/models/base.py", line 552, in get
entity = self._get_entity(id, host_path=host_path)
File "/opt/splunk/lib/python3.7/site-packages/splunk/models/base.py", line 468, in _get_entity
return self._fix_entity(splunk.entity.getEntity(self.model.resource, None, sessionKey=self.sessionKey, uri=id))
File "/opt/splunk/lib/python3.7/site-packages/splunk/entity.py", line 276, in getEntity
serverResponse, serverContent = rest.simpleRequest(uri, getargs=kwargs, sessionKey=sessionKey, raiseAllErrors=True)
File "/opt/splunk/lib/python3.7/site-packages/splunk/rest/init_.py", line 553, in simpleRequest
raise splunk.AuthenticationFailed
splunk.AuthenticationFailed: [HTTP 401] Client is not authenticated
2020-04-25 12:43:30,222+0000 INFO pid=18332 tid=MainThread file=configuration_check.py:run:299 | status="exiting" exit_status="2"
2020-04-25 12:44:30,066+0000 INFO pid=19331 tid=MainThread file=configuration_check.py::304 | status="starting"
2020-04-25 12:44:30,067+0000 INFO pid=19331 tid=MainThread file=configuration_check.py:run:129 | status="executing"