topic Re: Error msg="A script exited abnormally" status="exited with code 1" in All Apps and Add-ons

Error msg="A script exited abnormally" status="exited with code 1"

mcronkrite — Mon, 28 Sep 2020 17:23:46 GMT

On Nix system, still getting script errors AFTER doing chmod u+x for Splunk User on the Splunk_TA_Nix/bin/.sh files

msg="A script exited abnormally" input="./bin/cpu.sh" stanza="default" status="exited with code 1"
msg="A script exited abnormally" input="./bin/lsof.sh" stanza="default" status="exited with code 1"
msg="A script exited abnormally" input="./bin/rlog.sh" stanza="default" status="exited with code 1"
msg="A script exited abnormally" input="./bin/protocol.sh" stanza="default" status="exited with code 1"
msg="A script exited abnormally" input="./bin/bandwidth.sh" stanza="default" status="exited with code 1"
msg="A script exited abnormally" input="./bin/hardware.sh" stanza="default" status="exited with code 1"
msg="A script exited abnormally" input="./bin/sshdChecker.sh" stanza="default" status="exited with code 1"
msg="A script exited abnormally" input="./bin/iostat.sh" stanza="default" status="exited with code 1"
msg="A script exited abnormally" input="./bin/vmstat.sh" stanza="default" status="exited with code 1"
msg="A script exited abnormally" input="./bin/interfaces.sh" stanza="default" status="exited with code 1"

Re: Error msg="A script exited abnormally" status="exited with code 1"

Lowell — Tue, 25 Nov 2014 18:14:28 GMT

First off, doing a "chmod +x" makes the scripts executable which explain why this started showing up. (If these files are not executable, then Splunk cannot run them. Most often this is a packaging bug.)

So I just tracked down the error with rlog.sh. The following line is to blame:

awk -v START=$SEEK -v OUTPUT=$SEEK_FILE 'NR>START { print } END { print NR > OUTPUT }' $AUDIT_FILE | tee $TEE_DEST | /sbin/ausearch -i 2>/dev/null | grep -v "^----"

The problem is that grep will return a non-zero error message when it doesn't find what it's looking for. So in a normal case (on an active system), if new new events have been written to /var/log/audit/audit.log, then the line "---" will be written by ausearch and then grep will be happily remove it. (exit code 0). However, if there are no new events in audit.log then grep will be grumpy because it didn't have any work to do and exit with code 1. Since this is last line in the script to be executed, the script exits with grep's exit code, which is 1. Hence the error message you're seeing.

So the reality is, we don't care about grep's happiness, so a real simple solution is to just always report success by explicitly returning 0. That can be done like so:

awk -v START=$SEEK -v OUTPUT=$SEEK_FILE 'NR>START { print } END { print NR > OUTPUT }' $AUDIT_FILE | tee $TEE_DEST | /sbin/ausearch -i 2>/dev/null | grep -v "^----"
exit 0

I'm not sure what's happening in sshdChecker.sh, and it's certainly conceivable that you're hitting a different issue with rlog.sh but you really just have to dig in to the shell script to see what's going on. Of course if you don't need the input, just disable it and the annoying error will go away.

Update:

A slightly more efficent option would be to not even execute the command unless new content was added to the audit.log file. This can be done by adding the following:

if [ $FILE_LINES -eq $SEEK ] ; then
    # No new events in audit.log
    exit 0
fi

Re: Error msg="A script exited abnormally" status="exited with code 1"

jgoddard — Fri, 09 Oct 2015 17:56:32 GMT

Thank you for this. I had just dove deep enough to find that error and was scratching my head as to where the "division by zero" error was coming from.

No longer seeing the error.

Re: Error msg="A script exited abnormally" status="exited with code 1"

vsingla1 — Tue, 29 Dec 2015 19:16:21 GMT

Hi Lowell,
I am facing the issue but the solution you provided is not working.
I tried both the solutions:
1. Adding the IF loop that checks if new content was updated in the audit.log
2. Explicitly setting the exit code for AWK to zero.

None of them works and the script still shows the same error. Do you recommend anything else? I am running Splunk 6.2.3

Re: Error msg="A script exited abnormally" status="exited with code 1"

Lowell — Tue, 05 Jan 2016 23:11:21 GMT

What error are you seeing? What happens if you run the script manually?

Re: Error msg="A script exited abnormally" status="exited with code 1"

vsingla1 — Fri, 12 Feb 2016 20:30:08 GMT

This is the error.
msg="A script exited abnormally" input="./bin/rlog.sh" stanza="default" status="exited with code 1"

I thought the solution you provided will make these messages go away.
But since the rlog.sh is scheduled to kick-off every hour, these messages/error appear houlry on the Splunk search web User Interface.

Re: Error msg="A script exited abnormally" status="exited with code 1"

Lowell — Fri, 12 Feb 2016 23:39:37 GMT

Please try running the script from the command line (shell/terminal) manually and report back what the response is. If it doesn't run correctly from the command line, it's not going to work from Splunk either. I suspect that is the problem, but need more detail. Try something like this:

cd /opt/splunk/etc/apps/Splunk_TA_nix/bin/
./rlog.sh > /dev/null
echo "returncode=$?"

Re: Error msg="A script exited abnormally" status="exited with code 1"

vsingla1 — Mon, 15 Feb 2016 18:08:15 GMT

I ran the script and returncode=1

Re: Error msg="A script exited abnormally" status="exited with code 1"

mcronkrite — Wed, 30 Mar 2016 17:16:15 GMT

So turns out that if you run the scripts directly you can find out what the actually error is.
In my case the system was a new Centos 7 build and didn't have the following commands installed,
sar, netstat, etc etc.

The way to find it is by running this for each of the scripts not working.

bash -x ../bin/vmstat.sh

Re: Error msg="A script exited abnormally" status="exited with code 1"

vince2010091 — Wed, 09 Nov 2016 17:28:49 GMT

Hello,

On my Ubuntu i had to install SAR with :

apt install sysstat

Regards,
Vince

Re: Error msg="A script exited abnormally" status="exited with code 1"

salbro — Fri, 13 Jul 2018 15:12:40 GMT

This is still a valid fix with version 6.0.0 -- thank you!

Re: Error msg="A script exited abnormally" status="exited with code 1"

sloshburch — Wed, 16 Jan 2019 20:23:05 GMT

If it helps, I've built upon this topic with the What are best practices for deploying the Splunk Add-on for Unix and Linux in a distributed environment?

Re: Error msg="A script exited abnormally" status="exited with code 1"

jeburkes76 — Tue, 29 Sep 2020 23:04:47 GMT

rlog.sh
Lowell's file check for rlog.sh works well I believe, still testing.

sshdchecker
I had an error for sshdChecker.sh and in reviewing it I determined that if you are running Splunk as non-root user (best practice) it probably does not have read access on the sshd_config file so it errors out. Make sure the splunk service account has read access to the sshd_config file.