topic Re: Splunk universal forwarder fails to start - AIX in Getting Data In

Splunk universal forwarder fails to start - AIX

annebeate — Wed, 21 May 2014 06:05:54 GMT

Hi,

I installed the universal forwarder 6.1 for AIX.

splunkforwarder-6.1.1-207789-AIX-powerpc.tar

splunk@xxxxxx /opt/splunk > uname -a
AIX xxxxxx 1 6 00F783D04C00

The first time the forwarder was restarted I got this error:

splunk@xxxxxxxx /opt/splunk/splunkforwarder/bin > ./splunk start

Splunk> All batbelt. No tights.

Checking prerequisites...
Checking mgmt port [8089]: open
Checking conf files for problems...
Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
execve: Permission denied

I can't find anything about this in the splunk log files.

Best regards,
Anne Beate Arneberg

Re: Splunk universal forwarder fails to start - AIX

MuS — Wed, 21 May 2014 06:18:08 GMT

Hi annebeate,

check the permission of /opt/splunk, all the files must be owned by the user splunk in your case. If you started the forwarder before with the root account, you will have some files owned by root. Check the docs about Run Splunk as a different or non-root user

hope this helps ...

cheers, MuS

Re: Splunk universal forwarder fails to start - AIX

drdosia — Wed, 11 Jun 2014 19:32:19 GMT

We are running splunkforwarder-6.1.1-207789-AIX-powerpc.
We succesfully had it start and run for 2 day but today we get the exact same error as above.
We checked that all files and directories under .apps/splunk (where all our instances go) are owned by the correct splunk service account.

We are stumped. Any suggestions?

Re: Splunk universal forwarder fails to start - AIX

annebeate — Thu, 12 Jun 2014 05:52:35 GMT

Hi,

It did not help to change the file permissions to splunk. I tried to reinstall the 6.1.1 version and encountered the same problem during restart of splunk. I ended up with installing the 6.0.4 version which worked 🙂

Re: Splunk universal forwarder fails to start - AIX

HansWurscht — Fri, 22 Aug 2014 08:14:05 GMT

This will be fixed in 6.1.4

Re: Splunk universal forwarder fails to start - AIX

fantai_carefirs — Wed, 03 Sep 2014 16:59:25 GMT

What is the issue? I'm doing a pilot on my AIX server farm, and out of the 20 or so, one failed. It actually ran before. And now it's not running. This is very strange.

Re: Splunk universal forwarder fails to start - AIX

pit4getit — Thu, 18 Sep 2014 15:19:33 GMT

Hi,

did have same problem.
My problem was i did install first time the splunkforwarder with"splunk enable boot-start".
This makes entries in SRCsubsys of aix:

Check if entry exists:
odmget SRCsubsys | grep splunk

Get rid of entries with
sudo .../.splunk disable boot-start

Check if entry exists, and start splunk with wrapper script as user splunk, and never as user root like:

$ cat rc.splunk

#!/bin/ksh
### Script Variables
APPDIR=/usr/local/splunk/splunkforwarder/bin
APPNAME=splunk
RUN_USER=splunk

#export JAVA_HOME=/usr/java7_64

case "$1" in

start )
   echo "Starting the ${APPNAME} Application"
   su $RUN_USER -c ${APPDIR}/${APPNAME} start
;;

stop )
   echo "stopping the  Application"
   su $RUN_USER -c ${APPDIR}/${APPNAME} stop
;;

restart )
   echo "restarting  the ${APPNAME} Application"
   su $RUN_USER -c ${APPDIR}/${APPNAME} stop
   su $RUN_USER -c ${APPDIR}/${APPNAME} start
;;


* )
   echo "Usage: $0 COMMAND"
   echo "---------------------------"
   echo "start    - Start ${APPNAME} instance in ${APPDIR}"
   echo "stop     - Stop ${APPNAME} instance in ${APPDIR}"
   echo "restart     - Stop ${APPNAME} instance in ${APPDIR}"
   echo
   exit 1

esac

Hope thie helps

cheers, pit

Re: Splunk universal forwarder fails to start - AIX

vsingla1 — Wed, 01 Oct 2014 16:46:38 GMT

@HansWurscht
I am also facing same issue i.e execve: Permission denied when trying to run splunkforwarder on AIx.
I have tried splunkforwarder 6.1.2,6.1.3 and 6.1.4. But none of the versions have a fix for the issue. All of them still gives same error.

Re: Splunk universal forwarder fails to start - AIX

vsingla1 — Wed, 01 Oct 2014 18:43:46 GMT

Also, I installed 6.0.6 (last release for 6.0) splunkforwarder for AIX and it works fine.
Then I removed it and tried 6.1.0(first release for 6.1) version and it breaks there i.e the same execv:permission denied.
So it seems something happened while going from 6.0 to 6.1.

Re: Splunk universal forwarder fails to start - AIX

mgaraventa_splu — Mon, 28 Sep 2020 18:42:36 GMT

If you run into issues with Splunk starting on 6.1.x and 6.2.x versions please try following steps:

uninstall splunk.
As "root" run lssrc -a | grep splunk. if this returns any output, it means that we still have Splunk information in the object class. Typically this will be "splunkd". If this exists, it needs to be removed with rmssys -s splunkd .
Run lssrc -a | grep splunk to ensure that "splunkd" is removed from the object class.
Reboot the system.
As "root" untar to /opt .
As "root" run chmod -R splunk /opt/splunkforwarder. This will change the permissions of all the files under /opt/splunkforwarder to "splunk".
Check step 6 with the command find /opt/splunkforwarder \! -user splunk | xargs ls -al. This will return NO results if all the files are owned by splunk.
Edit /opt/splunkforwarder/etc/splunk-launch.conf and set SPLUNK_OS_USER=splunk .
Log out "root".
Login as "splunk" and run /opt/splunkforwarder/bin/splunk start. Splunk should now start.
You will need to login as "splunk" to stop and start splunk and to run any Splunk commands OR add splunk to /etc/sudoers and use sudo -u splunk.

There is actually a bug open for this issue: one for the 6.1.x releases (SPL-85092) and a clone for the 6.2.x releases (SPL-92963). The expected maintenance releases containing the fix are 6.1.7 and 6.2.2 respectively. Once they are released, please upgrade and make sure that you don't see issues anymore. If you should still face issues, then please file a new support case by providing:

A. a new diag
B. the history of actions perfomed with the according outputs.

and Splunk> Support will then further investigate the issue.

Re: Splunk universal forwarder fails to start - AIX

drdosia — Sat, 24 Jan 2015 02:27:26 GMT

We had the same issue with all versions of 6.1.x , 6.2 and the last tried was 6.2.1.
We resorted to replacing all of our startup scripts with Wrapper scripts like the one recommended by pit4getit .
That solves the issue for us but not a recommended final fix due to the extra work of replacing the startup scripts.

Re: Splunk universal forwarder fails to start - AIX

tweaktubbie — Wed, 07 Oct 2015 07:56:42 GMT

With these SPL's/bugs I had expected not to see this:

"execve: Permission denied
while running command /usr/bin/startsrc
Splunk boot-start is enabled. please use /usr/bin/startsrc -s splunkd to start splunk "

when I yesterday tried to migrate 6.0.5 UF's to 6.3.0. But the issue is still there. Is this something that was caused by the old installation, or is it still not fixed? It also occured in 6.1.1 and was said to be solved in 6.1.4 after fixing SPL-85092