Enterprise Security script exited abnormally status="exited with code 3"

[asimagu]

hi gents,

we are getting the following error in our search heads. any ideas about what can be happening?
I already checked the swap memory and it seems fine. We are running version 4.5.1 of ES now and this did not happen with previous versions.

Error:
msg="A script exited abnormally" input="/export/gcs1/data/splunk/etc/apps/SA-Utils/bin/configuration_check.py" stanza="configuration_check://confcheck_reload_auth" status="exited with code 3"

[pbugeja]

I added the following to the confcheck_es_app_version job, but still getting the error 3 messages.

ES Version 4.7.0

((streamfwd|splunk-(wmi.path|MonitorNoHandle.exe|winevtlog.exe|netmon.exe|perfmon.exe|regmon.exe|winprintmon.exe|admon.exe|powershell.exe)).*exited with code 1)|(confcheck_es_app_version.*exited with code 3)

Please let me know if you had any luck using the suppression field and post the command that was effective.

Thanks, Paul

[jgbricker]

Paul,

As @gjanders is hinting at with his question, the appropriate suppress syntax may change in terms of what is most appropriate for what you are trying to suppress. My error is worded slightly differently than the example given above. I have tried the following edit to my config at /opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/inputs.conf

[configuration_check://confcheck_es_app_version]
    interval = 86400
    default_severity = INFO
    required_ui_severity = WARN
    suppress = (.*data_migrator.py)
    debug = False

[gjanders]

Can you confirm which error message you get ?
This is a regular expression so you could just update it with any valid regular expression to ignore your message, the example I pasted is for confcheck_es_app_version exited with code 3...

[ehollima]

It's not fixed in 4.7.1...

Thanks,

Arion

[gjanders]

Splunk support provided this workaround:

1. Disable the configuration_check://confcheck_es_app_version input (OR)
2. Add confcheck_es_app_version to the suppress of configuration_check://confcheck_script_errors. Something like suppress = ((streamfwd|splunk-(wmi.path|MonitorNoHandle.exe|winevtlog.exe|netmon.exe|perfmon.exe|regmon.exe|winprintmon.exe|admon.exe|powershell.exe)).*exited with code 1)|(confcheck_es_app_version.*exited with code 3) The confcheck settings are located in $SPLUNK_HOME/etc/apps/SplunkEnterpriseSecuritySuite/default/inputs.conf , just override in your local copy of it...

[hcannon]

In case it's useful to anyone else, the [configuration_check://confcheck_es_app_version] input is located here:
$SPLUNK_HOME/etc/apps/SplunkEnterpriseSecuritySuite/

[brucejohnson]

I ended up commenting out the specific configuration check in $SPLUNK_HOME/etc/apps/SplunkEnterpriseSecuritySuite/default/inputs.conf. It's the first time I have edited a default file but I did it because presumably it will be fixed by the next time I install / upgrade ES and I want it to be overlaid. If I override it in local or suppress it, I'll forget that I did. Presumably the configuration check is there for a reason.

[gjanders]

Thanks, updated the comment.

[jwelch_splunk]

Can you show me the results of:

find /opt/splunk/etc -name inputs.conf |xargs grep reload_auth |grep -v old

Pretty sure this was pulled, and I am wondering if you have a local over-ride.

Okie

[asimagu]

hi, thanks for your help jwelch

This is the output of that command:

find: ‘/opt/splunk/etc/users/losanic/search’: Permission denied
find: ‘/opt/splunk/etc/users/kundjas/SplunkEnterpriseSecuritySuite’: Permission denied
find: ‘/opt/splunk/etc/users/aguaang/search/history’: Permission denied
find: ‘/opt/splunk/etc/users/aguaang/user-prefs’: Permission denied
find: ‘/opt/splunk/etc/users/aguaang/SplunkEnterpriseSecuritySuite’: Permission denied
find: ‘/opt/splunk/etc/users/aguaang/SA-ThreatIntelligence’: Permission denied
/opt/splunk/etc/apps/SA-ThreatIntelligence/local/inputs.conf:[configuration_check://confcheck_reload_auth]
/opt/splunk/etc/apps/SA-ThreatIntelligence/default/inputs.conf:[configuration_check://confcheck_reload_auth]

[jwelch_splunk]

I don't see this inputs on my current version any longer. I am looking for confirmation that we did in fact pull it out at some point.

My thoughts are your upgrade possibly failed:

paste results of:

tail -10 /opt/splunk/var/log/splunk/essinstaller2.log

[asimagu]

no results. That logfile exists but it´s empty

[jwelch_splunk]

Did you do an upgrade recently?

[asimagu]

Professional Services from Splunk did it in late November.

[asimagu]

do you recommend that I open a support case?

[jwelch_splunk]

yes give me the case number when you do and I will assign it to myself and reach out to you

[asimagu]

Support Case: 465446

[rgaube]

We experience the same issue ... can you please be so kind and share the outcome and how to solve this issue? Thanks!

[jwelch_splunk]

Are you sure this is the same issue? Did you recently upgrade to 4.7.0? If so let me know.

[mikaelbje]

I'm also getting

msg="A script exited abnormally" input="/opt/splunk/etc/apps/SA-Utils/bin/configuration_check.py" stanza="configuration_check://confcheck_es_app_version" status="exited with code 3"

after a recent upgrade to 4.7.0 on Splunk 6.6.0