From the Monitoring Console:
Health Check: msg="A script exited abnormally with exit status: 4"
Health Check: msg="A script exited abnormally with exit status: 1" input="./opt/splunk/etc/apps/threatstream/bin/tsiocingest.py" stanza="tsiocingest://threatstream_app"
Working on resolving this health check warning occurring on our ES instance. Has anyone experienced it before?
There might be possibility that those modular input failed once, but next time it ran successfully. To check latest status of all modular inputs, please use below query and look at
| rest /services/admin/inputstatus/ModularInputs:modular%20input%20commands splunk_server=local count=0 | append [| rest /services/admin/inputstatus/ExecProcessor:exec%20commands splunk_server=local count=0] | fields inputs* | transpose | rex field=column "inputs(?<script>\S+)(?:\s\((?<stanza>[^\(]+)\))?\.(?<key>(exit status description)|(time closed)|(time opened))" | eval value=coalesce('row 1', 'row 2'), stanza=coalesce(stanza, "default"), started=if(key=="time opened", value, started), stopped=if(key=="time closed", value, stopped) | rex field=value "exited\s+with\s+code\s+(?<exit_status>\d+)" | stats first(started) as started, first(stopped) as stopped, first(exit_status) as exit_status by script, stanza | eval errmsg=case(exit_status=="0", null(), isnotnull(exit_status), "A script exited abnormally with exit status: "+exit_status, isnull(started) or isnotnull(stopped), "A script is in an unknown state"), ignore=if(`script_error_msg_ignore`, 1, 0)
There is schedule search
Audit - Script Errors in SA_Utils app which run at every hour and check whether
errmsg is not null and ignore not equal to 0 in above search output then it will fire custom alert action which will write error message in Splunk Web top right corner under Messages.
If one or more modular inputs failed once and
Audit - Script Errors ingested data in Messages and next time if Modular input works fine then older messages under
Messages in Splunk Web will not be deleted automatically and you need to delete it manually. So you need to check whether modular inputs failing continously? If yes then you can check why script is failing in
$SPLUNK_HOME/var/log/splunk/threatlist.log on Search Head.
Thanks for the assist, @harsmarvania57. Turned out to be a temporary issue with our Anomali feed.