Error msg="A script exited abnormally" status="exited with code 1"

[mcronkrite]

On Nix system, still getting script errors AFTER doing chmod u+x for Splunk User on the Splunk_TA_Nix/bin/.sh files

msg="A script exited abnormally" input="./bin/cpu.sh" stanza="default" status="exited with code 1"
msg="A script exited abnormally" input="./bin/lsof.sh" stanza="default" status="exited with code 1"
msg="A script exited abnormally" input="./bin/rlog.sh" stanza="default" status="exited with code 1"
msg="A script exited abnormally" input="./bin/protocol.sh" stanza="default" status="exited with code 1"
msg="A script exited abnormally" input="./bin/bandwidth.sh" stanza="default" status="exited with code 1"
msg="A script exited abnormally" input="./bin/hardware.sh" stanza="default" status="exited with code 1"
msg="A script exited abnormally" input="./bin/sshdChecker.sh" stanza="default" status="exited with code 1"
msg="A script exited abnormally" input="./bin/iostat.sh" stanza="default" status="exited with code 1"
msg="A script exited abnormally" input="./bin/vmstat.sh" stanza="default" status="exited with code 1"
msg="A script exited abnormally" input="./bin/interfaces.sh" stanza="default" status="exited with code 1"

[vince2010091]

Hello,

On my Ubuntu i had to install SAR with :

apt install sysstat

Regards,
Vince

[mcronkrite · Accepted solution]

So turns out that if you run the scripts directly you can find out what the actually error is.
In my case the system was a new Centos 7 build and didn't have the following commands installed,
sar, netstat, etc etc.

The way to find it is by running this for each of the scripts not working.

bash -x ../bin/vmstat.sh

[sloshburch]

If it helps, I've built upon this topic with the What are best practices for deploying the Splunk Add-on for Unix and Linux in a distributed environm...

[Lowell]

First off, doing a "chmod +x" makes the scripts executable which explain why this started showing up. (If these files are not executable, then Splunk cannot run them. Most often this is a packaging bug.)

So I just tracked down the error with rlog.sh. The following line is to blame:

awk -v START=$SEEK -v OUTPUT=$SEEK_FILE 'NR>START { print } END { print NR > OUTPUT }' $AUDIT_FILE | tee $TEE_DEST | /sbin/ausearch -i 2>/dev/null | grep -v "^----"

The problem is that grep will return a non-zero error message when it doesn't find what it's looking for. So in a normal case (on an active system), if new new events have been written to /var/log/audit/audit.log, then the line "---" will be written by ausearch and then grep will be happily remove it. (exit code 0). However, if there are no new events in audit.log then grep will be grumpy because it didn't have any work to do and exit with code 1. Since this is last line in the script to be executed, the script exits with grep's exit code, which is 1. Hence the error message you're seeing.

So the reality is, we don't care about grep's happiness, so a real simple solution is to just always report success by explicitly returning 0. That can be done like so:

awk -v START=$SEEK -v OUTPUT=$SEEK_FILE 'NR>START { print } END { print NR > OUTPUT }' $AUDIT_FILE | tee $TEE_DEST | /sbin/ausearch -i 2>/dev/null | grep -v "^----"
exit 0

I'm not sure what's happening in sshdChecker.sh, and it's certainly conceivable that you're hitting a different issue with rlog.sh but you really just have to dig in to the shell script to see what's going on. Of course if you don't need the input, just disable it and the annoying error will go away.

Update:

A slightly more efficent option would be to not even execute the command unless new content was added to the audit.log file. This can be done by adding the following:

if [ $FILE_LINES -eq $SEEK ] ; then
    # No new events in audit.log
    exit 0
fi

[jeburkes76]

rlog.sh
Lowell's file check for rlog.sh works well I believe, still testing.

sshdchecker
I had an error for sshdChecker.sh and in reviewing it I determined that if you are running Splunk as non-root user (best practice) it probably does not have read access on the sshd_config file so it errors out. Make sure the splunk service account has read access to the sshd_config file.

[salbro]

This is still a valid fix with version 6.0.0 -- thank you!

[vsingla1]

Hi Lowell,
I am facing the issue but the solution you provided is not working.
I tried both the solutions:
1. Adding the IF loop that checks if new content was updated in the audit.log
2. Explicitly setting the exit code for AWK to zero.

None of them works and the script still shows the same error. Do you recommend anything else? I am running Splunk 6.2.3

[Lowell]

What error are you seeing? What happens if you run the script manually?

[vsingla1]

This is the error.
msg="A script exited abnormally" input="./bin/rlog.sh" stanza="default" status="exited with code 1"

I thought the solution you provided will make these messages go away.
But since the rlog.sh is scheduled to kick-off every hour, these messages/error appear houlry on the Splunk search web User Interface.

[Lowell]

Please try running the script from the command line (shell/terminal) manually and report back what the response is. If it doesn't run correctly from the command line, it's not going to work from Splunk either. I suspect that is the problem, but need more detail. Try something like this:

cd /opt/splunk/etc/apps/Splunk_TA_nix/bin/
./rlog.sh > /dev/null
echo "returncode=$?"

[vsingla1]

I ran the script and returncode=1

[jgoddard]

Thank you for this. I had just dove deep enough to find that error and was scratching my head as to where the "division by zero" error was coming from.

No longer seeing the error.