Getting Data In

Splunk universal forwarder fails to start - AIX

Path Finder


I installed the universal forwarder 6.1 for AIX.


splunk@xxxxxx /opt/splunk > uname -a
AIX xxxxxx 1 6 00F783D04C00

The first time the forwarder was restarted I got this error:

splunk@xxxxxxxx /opt/splunk/splunkforwarder/bin > ./splunk start

Splunk> All batbelt. No tights.

Checking prerequisites...
Checking mgmt port [8089]: open
Checking conf files for problems...
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
execve: Permission denied

I can't find anything about this in the splunk log files.

Best regards,
Anne Beate Arneberg

Tags (1)


We had the same issue with all versions of 6.1.x , 6.2 and the last tried was 6.2.1.
We resorted to replacing all of our startup scripts with Wrapper scripts like the one recommended by pit4getit .
That solves the issue for us but not a recommended final fix due to the extra work of replacing the startup scripts.

0 Karma

Splunk Employee
Splunk Employee

If you run into issues with Splunk starting on 6.1.x and 6.2.x versions please try following steps:

  1. uninstall splunk.
  2. As "root" run lssrc -a | grep splunk. if this returns any output, it means that we still have Splunk information in the object class. Typically this will be "splunkd". If this exists, it needs to be removed with rmssys -s splunkd .
  3. Run lssrc -a | grep splunk to ensure that "splunkd" is removed from the object class.
  4. Reboot the system.
  5. As "root" untar to /opt .
  6. As "root" run chmod -R splunk /opt/splunkforwarder. This will change the permissions of all the files under /opt/splunkforwarder to "splunk".
  7. Check step 6 with the command find /opt/splunkforwarder \! -user splunk | xargs ls -al. This will return NO results if all the files are owned by splunk.
  8. Edit /opt/splunkforwarder/etc/splunk-launch.conf and set SPLUNK_OS_USER=splunk .
  9. Log out "root".
  10. Login as "splunk" and run /opt/splunkforwarder/bin/splunk start. Splunk should now start.
  11. You will need to login as "splunk" to stop and start splunk and to run any Splunk commands OR add splunk to /etc/sudoers and use sudo -u splunk.

There is actually a bug open for this issue: one for the 6.1.x releases (SPL-85092) and a clone for the 6.2.x releases (SPL-92963). The expected maintenance releases containing the fix are 6.1.7 and 6.2.2 respectively. Once they are released, please upgrade and make sure that you don't see issues anymore. If you should still face issues, then please file a new support case by providing:

A. a new diag
B. the history of actions perfomed with the according outputs.

and Splunk> Support will then further investigate the issue.


With these SPL's/bugs I had expected not to see this:

"execve: Permission denied
while running command /usr/bin/startsrc
Splunk boot-start is enabled. please use /usr/bin/startsrc -s splunkd to start splunk "

when I yesterday tried to migrate 6.0.5 UF's to 6.3.0. But the issue is still there. Is this something that was caused by the old installation, or is it still not fixed? It also occured in 6.1.1 and was said to be solved in 6.1.4 after fixing SPL-85092

0 Karma


Also, I installed 6.0.6 (last release for 6.0) splunkforwarder for AIX and it works fine.
Then I removed it and tried 6.1.0(first release for 6.1) version and it breaks there i.e the same execv:permission denied.
So it seems something happened while going from 6.0 to 6.1.


I am also facing same issue i.e execve: Permission denied when trying to run splunkforwarder on AIx.
I have tried splunkforwarder 6.1.2,6.1.3 and 6.1.4. But none of the versions have a fix for the issue. All of them still gives same error.

0 Karma



did have same problem.
My problem was i did install first time the splunkforwarder with"splunk enable boot-start".
This makes entries in SRCsubsys of aix:

Check if entry exists:
odmget SRCsubsys | grep splunk

Get rid of entries with
sudo .../.splunk disable boot-start

Check if entry exists, and start splunk with wrapper script as user splunk, and never as user root like:

$ cat rc.splunk

### Script Variables

#export JAVA_HOME=/usr/java7_64

case "$1" in

start )
   echo "Starting the ${APPNAME} Application"
   su $RUN_USER -c ${APPDIR}/${APPNAME} start

stop )
   echo "stopping the  Application"
   su $RUN_USER -c ${APPDIR}/${APPNAME} stop

restart )
   echo "restarting  the ${APPNAME} Application"
   su $RUN_USER -c ${APPDIR}/${APPNAME} stop
   su $RUN_USER -c ${APPDIR}/${APPNAME} start

* )
   echo "Usage: $0 COMMAND"
   echo "---------------------------"
   echo "start    - Start ${APPNAME} instance in ${APPDIR}"
   echo "stop     - Stop ${APPNAME} instance in ${APPDIR}"
   echo "restart     - Stop ${APPNAME} instance in ${APPDIR}"
   exit 1


Hope thie helps

cheers, pit


What is the issue? I'm doing a pilot on my AIX server farm, and out of the 20 or so, one failed. It actually ran before. And now it's not running. This is very strange.

0 Karma

Path Finder

This will be fixed in 6.1.4

0 Karma

Path Finder


It did not help to change the file permissions to splunk. I tried to reinstall the 6.1.1 version and encountered the same problem during restart of splunk. I ended up with installing the 6.0.4 version which worked 🙂

0 Karma


We are running splunkforwarder-6.1.1-207789-AIX-powerpc.
We succesfully had it start and run for 2 day but today we get the exact same error as above.
We checked that all files and directories under .apps/splunk (where all our instances go) are owned by the correct splunk service account.

We are stumped. Any suggestions?

0 Karma


Hi annebeate,

check the permission of /opt/splunk, all the files must be owned by the user splunk in your case. If you started the forwarder before with the root account, you will have some files owned by root. Check the docs about Run Splunk as a different or non-root user

hope this helps ...

cheers, MuS

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!