Health Check: msg="A script exited abnormally with exit status: 4"

[dillardo_2]

From the Monitoring Console:

Health Check: msg="A script exited abnormally with exit status: 4"
input="./opt/splunk/etc/apps/DA-ESS-ThreatIntelligence/bin/threat_intelligence_manager.py" stanza="default"

Health Check: msg="A script exited abnormally with exit status: 1" input="./opt/splunk/etc/apps/threatstream/bin/ts_ioc_ingest.py" stanza="ts_ioc_ingest://threatstream_app"

Working on resolving this health check warning occurring on our ES instance. Has anyone experienced it before?

[dillardo_2]

Thanks for the assist, @harsmarvania57. Turned out to be a temporary issue with our Anomali feed.

[harsmarvania57 · Accepted solution]

Hi,

There might be possibility that those modular input failed once, but next time it ran successfully. To check latest status of all modular inputs, please use below query and look at exit_status

| rest /services/admin/inputstatus/ModularInputs:modular%20input%20commands splunk_server=local count=0 
| append [| rest /services/admin/inputstatus/ExecProcessor:exec%20commands splunk_server=local count=0] 
| fields inputs*
| transpose
| rex field=column "inputs(?<script>\S+)(?:\s\((?<stanza>[^\(]+)\))?\.(?<key>(exit status description)|(time closed)|(time opened))"
| eval value=coalesce('row 1', 'row 2'), stanza=coalesce(stanza, "default"), started=if(key=="time opened", value, started), stopped=if(key=="time closed", value, stopped)
| rex field=value "exited\s+with\s+code\s+(?<exit_status>\d+)"
| stats first(started) as started, first(stopped) as stopped, first(exit_status) as exit_status by script, stanza
| eval errmsg=case(exit_status=="0", null(), isnotnull(exit_status), "A script exited abnormally with exit status: "+exit_status, isnull(started) or isnotnull(stopped), "A script is in an unknown state"), ignore=if(`script_error_msg_ignore`, 1, 0)

There is schedule search Audit - Script Errors in SA_Utils app which run at every hour and check whether errmsg is not null and ignore not equal to 0 in above search output then it will fire custom alert action which will write error message in Splunk Web top right corner under Messages.

If one or more modular inputs failed once and Audit - Script Errors ingested data in Messages and next time if Modular input works fine then older messages under Messages in Splunk Web will not be deleted automatically and you need to delete it manually. So you need to check whether modular inputs failing continously? If yes then you can check why script is failing in $SPLUNK_HOME/var/log/splunk/threat_intelligence_manager.log and $SPLUNK_HOME/var/log/splunk/threatlist.log on Search Head.

[jawaharas]

Can you share a screenshot with the errors?