When trying to start Splunk, I'm getting an "execve: Permission denied" error. How can I fix this?

[DerekB]

Trying to start Splunk but getting an "execve: Permission denied " error

This is Splunk 6.1.x and my OS is AIX.

bin/splunk start --accept-license

Checking prerequisites...
WARNING: File size limit (ulimit -f) is set low (1073741312 bytes) Splunk may not work.
You may want to run "ulimit -f unlimited" before starting splunk.
Checking mgmt port [8089]: open
Creating: /opt/splunkforwarder/var/lib/splunk
Creating: /opt/splunkforwarder/var/run/splunk
Creating: /opt/splunkforwarder/var/run/splunk/appserver/i18n
Creating: /opt/splunkforwarder/var/run/splunk/appserver/modules/static/css
Creating: /opt/splunkforwarder/var/run/splunk/upload
Creating: /opt/splunkforwarder/var/spool/splunk
Creating: /opt/splunkforwarder/var/spool/dirmoncache
Creating: /opt/splunkforwarder/var/lib/splunk/authDb
Creating: /opt/splunkforwarder/var/lib/splunk/hashDb
New certs have been generated in '/opt/splunkforwarder/etc/auth'.
Checking conf files for problems...
Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
execve: Permission denied

[DerekB · Accepted solution]

The problem here is that splunk was configured with boot start enabled. On AIX, this will put a line in the /etc/inittab file which will then be run by startsrc. This seems fine except startsrc is run by root and if your Splunk install is configured to run as a different user, you can get a "permission denied" error.

To fix this, remove the line in /etc/inittab that references Splunk and restart.

If you do want Splunk to start at boot time, you could configure it with the "enable boot start" but just make sure to set the user Splunk will run as correctly. This doc page is a good reference.

http://docs.splunk.com/Documentation/Splunk/6.1.3/admin/ConfigureSplunktostartatboottime