Getting Data In

Splunk universal forwarder fails to start - AIX

annebeate
Path Finder

Hi,

I installed the universal forwarder 6.1 for AIX.

splunkforwarder-6.1.1-207789-AIX-powerpc.tar

splunk@xxxxxx /opt/splunk > uname -a
AIX xxxxxx 1 6 00F783D04C00

The first time the forwarder was restarted I got this error:

splunk@xxxxxxxx /opt/splunk/splunkforwarder/bin > ./splunk start

Splunk> All batbelt. No tights.

Checking prerequisites...
Checking mgmt port [8089]: open
Checking conf files for problems...
Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
execve: Permission denied

I can't find anything about this in the splunk log files.

Best regards,
Anne Beate Arneberg

Tags (1)

drdosia
Explorer

We had the same issue with all versions of 6.1.x , 6.2 and the last tried was 6.2.1.
We resorted to replacing all of our startup scripts with Wrapper scripts like the one recommended by pit4getit .
That solves the issue for us but not a recommended final fix due to the extra work of replacing the startup scripts.

0 Karma

mgaraventa_splu
Splunk Employee
Splunk Employee

If you run into issues with Splunk starting on 6.1.x and 6.2.x versions please try following steps:

  1. uninstall splunk.
  2. As "root" run lssrc -a | grep splunk. if this returns any output, it means that we still have Splunk information in the object class. Typically this will be "splunkd". If this exists, it needs to be removed with rmssys -s splunkd .
  3. Run lssrc -a | grep splunk to ensure that "splunkd" is removed from the object class.
  4. Reboot the system.
  5. As "root" untar to /opt .
  6. As "root" run chmod -R splunk /opt/splunkforwarder. This will change the permissions of all the files under /opt/splunkforwarder to "splunk".
  7. Check step 6 with the command find /opt/splunkforwarder \! -user splunk | xargs ls -al. This will return NO results if all the files are owned by splunk.
  8. Edit /opt/splunkforwarder/etc/splunk-launch.conf and set SPLUNK_OS_USER=splunk .
  9. Log out "root".
  10. Login as "splunk" and run /opt/splunkforwarder/bin/splunk start. Splunk should now start.
  11. You will need to login as "splunk" to stop and start splunk and to run any Splunk commands OR add splunk to /etc/sudoers and use sudo -u splunk.

There is actually a bug open for this issue: one for the 6.1.x releases (SPL-85092) and a clone for the 6.2.x releases (SPL-92963). The expected maintenance releases containing the fix are 6.1.7 and 6.2.2 respectively. Once they are released, please upgrade and make sure that you don't see issues anymore. If you should still face issues, then please file a new support case by providing:

A. a new diag
B. the history of actions perfomed with the according outputs.

and Splunk> Support will then further investigate the issue.

tweaktubbie
Communicator

With these SPL's/bugs I had expected not to see this:

"execve: Permission denied
while running command /usr/bin/startsrc
Splunk boot-start is enabled. please use /usr/bin/startsrc -s splunkd to start splunk "

when I yesterday tried to migrate 6.0.5 UF's to 6.3.0. But the issue is still there. Is this something that was caused by the old installation, or is it still not fixed? It also occured in 6.1.1 and was said to be solved in 6.1.4 after fixing SPL-85092

0 Karma

vsingla1
Communicator

Also, I installed 6.0.6 (last release for 6.0) splunkforwarder for AIX and it works fine.
Then I removed it and tried 6.1.0(first release for 6.1) version and it breaks there i.e the same execv:permission denied.
So it seems something happened while going from 6.0 to 6.1.

vsingla1
Communicator

@HansWurscht
I am also facing same issue i.e execve: Permission denied when trying to run splunkforwarder on AIx.
I have tried splunkforwarder 6.1.2,6.1.3 and 6.1.4. But none of the versions have a fix for the issue. All of them still gives same error.

0 Karma

pit4getit
Explorer

Hi,

did have same problem.
My problem was i did install first time the splunkforwarder with"splunk enable boot-start".
This makes entries in SRCsubsys of aix:

Check if entry exists:
odmget SRCsubsys | grep splunk

Get rid of entries with
sudo .../.splunk disable boot-start

Check if entry exists, and start splunk with wrapper script as user splunk, and never as user root like:

$ cat rc.splunk

#!/bin/ksh
### Script Variables
APPDIR=/usr/local/splunk/splunkforwarder/bin
APPNAME=splunk
RUN_USER=splunk

#export JAVA_HOME=/usr/java7_64

case "$1" in

start )
   echo "Starting the ${APPNAME} Application"
   su $RUN_USER -c ${APPDIR}/${APPNAME} start
;;

stop )
   echo "stopping the  Application"
   su $RUN_USER -c ${APPDIR}/${APPNAME} stop
;;

restart )
   echo "restarting  the ${APPNAME} Application"
   su $RUN_USER -c ${APPDIR}/${APPNAME} stop
   su $RUN_USER -c ${APPDIR}/${APPNAME} start
;;


* )
   echo "Usage: $0 COMMAND"
   echo "---------------------------"
   echo "start    - Start ${APPNAME} instance in ${APPDIR}"
   echo "stop     - Stop ${APPNAME} instance in ${APPDIR}"
   echo "restart     - Stop ${APPNAME} instance in ${APPDIR}"
   echo
   exit 1

esac

Hope thie helps

cheers, pit

fantai_carefirs
Explorer

What is the issue? I'm doing a pilot on my AIX server farm, and out of the 20 or so, one failed. It actually ran before. And now it's not running. This is very strange.

0 Karma

HansWurscht
Path Finder

This will be fixed in 6.1.4

0 Karma

annebeate
Path Finder

Hi,

It did not help to change the file permissions to splunk. I tried to reinstall the 6.1.1 version and encountered the same problem during restart of splunk. I ended up with installing the 6.0.4 version which worked 🙂

0 Karma

drdosia
Explorer

We are running splunkforwarder-6.1.1-207789-AIX-powerpc.
We succesfully had it start and run for 2 day but today we get the exact same error as above.
We checked that all files and directories under .apps/splunk (where all our instances go) are owned by the correct splunk service account.

We are stumped. Any suggestions?

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi annebeate,

check the permission of /opt/splunk, all the files must be owned by the user splunk in your case. If you started the forwarder before with the root account, you will have some files owned by root. Check the docs about Run Splunk as a different or non-root user

hope this helps ...

cheers, MuS

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...