Splunk Enterprise Security
Highlighted

Health Check: msg="A script exited abnormally with exit status: 4"

Path Finder

From the Monitoring Console:

Health Check: msg="A script exited abnormally with exit status: 4"
input="./opt/splunk/etc/apps/DA-ESS-ThreatIntelligence/bin/threatintelligencemanager.py" stanza="default"

Health Check: msg="A script exited abnormally with exit status: 1" input="./opt/splunk/etc/apps/threatstream/bin/tsiocingest.py" stanza="tsiocingest://threatstream_app"

Working on resolving this health check warning occurring on our ES instance. Has anyone experienced it before?

0 Karma
Highlighted

Re: Health Check: msg="A script exited abnormally with exit status: 4"

Motivator

Can you share a screenshot with the errors?

0 Karma
Highlighted

Re: Health Check: msg="A script exited abnormally with exit status: 4"

SplunkTrust
SplunkTrust

Hi,

There might be possibility that those modular input failed once, but next time it ran successfully. To check latest status of all modular inputs, please use below query and look at exit_status

| rest /services/admin/inputstatus/ModularInputs:modular%20input%20commands splunk_server=local count=0 
| append [| rest /services/admin/inputstatus/ExecProcessor:exec%20commands splunk_server=local count=0] 
| fields inputs*
| transpose
| rex field=column "inputs(?<script>\S+)(?:\s\((?<stanza>[^\(]+)\))?\.(?<key>(exit status description)|(time closed)|(time opened))"
| eval value=coalesce('row 1', 'row 2'), stanza=coalesce(stanza, "default"), started=if(key=="time opened", value, started), stopped=if(key=="time closed", value, stopped)
| rex field=value "exited\s+with\s+code\s+(?<exit_status>\d+)"
| stats first(started) as started, first(stopped) as stopped, first(exit_status) as exit_status by script, stanza
| eval errmsg=case(exit_status=="0", null(), isnotnull(exit_status), "A script exited abnormally with exit status: "+exit_status, isnull(started) or isnotnull(stopped), "A script is in an unknown state"), ignore=if(`script_error_msg_ignore`, 1, 0)

There is schedule search Audit - Script Errors in SA_Utils app which run at every hour and check whether errmsg is not null and ignore not equal to 0 in above search output then it will fire custom alert action which will write error message in Splunk Web top right corner under Messages.

If one or more modular inputs failed once and Audit - Script Errors ingested data in Messages and next time if Modular input works fine then older messages under Messages in Splunk Web will not be deleted automatically and you need to delete it manually. So you need to check whether modular inputs failing continously? If yes then you can check why script is failing in $SPLUNK_HOME/var/log/splunk/threat_intelligence_manager.log and $SPLUNK_HOME/var/log/splunk/threatlist.log on Search Head.

View solution in original post

Highlighted

Re: Health Check: msg="A script exited abnormally with exit status: 4"

Path Finder

Thanks for the assist, @harsmarvania57. Turned out to be a temporary issue with our Anomali feed.

0 Karma