When installing the Splunk 6.1.1 Universal Forwarder on AIX7.1, splunkd
seems to require read access on /etc/inittab
or else the daemon will fail to start. We run the Splunk daemon under a limited (non-root) useraccount and the inittab
file is protected from unauthorized access.
Steps to reproduce:
[root@bdaix71]:/tmp # su - splunk
[splunk@bdaix71]:/opt/splunk $ uf/bin/splunk version
Splunk Universal Forwarder 6.1.1 (build 207789)
[splunk@bdaix71]:/opt/splunk $ uf/bin/splunk status
splunkd is not running.
[splunk@bdaix71]:/opt/splunk $ uf/bin/splunk start
Splunk> CSI: Logfiles.
Checking prerequisites...
Checking mgmt port [8089]: open
Checking conf files for problems...
Done
All preliminary checks passed.
Starting splunk server daemon (splunkd)...
reading /etc/inittab: Permission denied
[splunk@bdaix71]:/opt/splunk $ uf/bin/splunk status
splunkd is not running.
Permissions on /etc/inittab
:
[root@bdaix71]:/tmp # ls -la /etc/inittab
-rw------- 1 root system 3155 Feb 24 16:42 /etc/inittab
After changing /etc/inittab
to group/worldreadable:
[root@bdaix71]:/tmp # chmod 644 /etc/inittab
[root@bdaix71]:/tmp # su - splunk
[splunk@bdaix71]:/opt/splunk $ uf/bin/splunk status
splunkd is not running.
[splunk@bdaix71]:/opt/splunk $ uf/bin/splunk start
Splunk> CSI: Logfiles.
Checking prerequisites...
Checking mgmt port [8089]: open
Checking conf files for problems...
Done
All preliminary checks passed.
Starting splunk server daemon (splunkd)...
Declared role=universal_forwarder.
Done
[splunk@bdaix71]:/opt/splunk $ uf/bin/splunk status
splunkd is running (PID: 7536766).
splunk helpers are running (PIDs: 5111876).
Starting splunk
in --debug
mode does not reveal extra information regarding the /etc/inittab
file. Logging in var/log/splunk/
does not mention the inittab
file and grepping for inittab
only lists references in the splunkd
binary.
Read access on this file seems pointless and is probably only required when the sysadmin wants to use the autostart capabilities(?).
we have the same problem with UF version 6.2.2-255606.
does anyone have a fix for it?
Thanks.
After simply ignoring this issue for quite a while, it has become relevant again, since we are currently in the process of upgrading from Splunk v5 to v6. One of my colleagues found a workaround which appears to partially solve our problems. I am posting this workaround for future reference and peer review.
1: Enable SRC subsystem
First, enable the Splunk SRC subsystem in AIX and instruct it to use the non-privileged useraccount (we'll use splunk
, in this case):
/opt/splunk/uf/bin/splunk enable boot-start -user splunk
2: Remove SPLUNK_OS_USER
limitation
Now, you need to remove the SPLUNK_OS_USER
line in splunk-launch.conf
. This will be added automatically by the previous command. If you don't, you won't be able to run the splunk start
command as a user other than the one you specified earlier (and this will still fail due to the root cause of the inittab
problem not being solved).
That's it. You will now be able to either use the default AIX startsrc -g splunk
and stopsrc -g splunk
commands and even run the binary as the root
user. The daemon will drop privileges to the previously specified useraccount.
What you (still) won't be able to do
You still won't be able to run the splunk start
and splunk stop
commands directly as the non-privileged account. This may be a problem for some.
Testcases
After performing the actions in 1 and 2, the following testcases were carried out sequentially:
Starting point, Splunk is not running:
[root@bdaix71]:/root # oslevel -s
7100-03-01-1341
[root@bdaix71]:/root # /opt/splunk/uf/bin/splunk version
Splunk Universal Forwarder 6.2.1 (build 245427)
[root@bdaix71]:/root # ps -ef|grep [s]plunk
(no output)
[root@bdaix71]:/root # lssrc -g splunk
Subsystem Group PID Status
splunkd splunk inoperative
Start Splunk as root
using the AIX SRC subsystem:
[root@bdaix71]:/root # startsrc -g splunk
0513-059 The splunkd Subsystem has been started. Subsystem PID is 4849824.
[root@bdaix71]:/root # ps -ef|grep [s]plunk
splunk 4849824 1900760 0 12:19:37 - 0:00 /opt/splunk/uf/bin/splunk _internal_exec_splunkd
splunk 11141354 4849824 120 12:19:42 - 0:00 /opt/splunk/uf/bin/splunkd btool server list general --no-log
[root@bdaix71]:/root # lssrc -g splunk
Subsystem Group PID Status
splunkd splunk 4849824 active
Stop Splunk as root
using the AIX SRC subsystem:
[root@bdaix71]:/root # stopsrc -g splunk
0513-044 The splunkd Subsystem was requested to stop.
[root@bdaix71]:/root # lssrc -g splunk
Subsystem Group PID Status
splunkd splunk inoperative
[root@bdaix71]:/root # ps -ef|grep [s]plunk
(no output)
Start Splunk as root
using direct call to binary:
[root@bdaix71]:/root # /opt/splunk/uf/bin/splunk start
Splunk> CSI: Logfiles.
Checking prerequisites...
Checking mgmt port [8089]: open
Checking conf files for problems...
Done
All preliminary checks passed.
Starting splunk server daemon (splunkd)...
0513-059 The splunkd Subsystem has been started. Subsystem PID is 13762732.
Done
[root@bdaix71]:/root # ps -ef|grep [s]plunk
splunk 8781916 13762732 0 12:23:31 - 0:00 [splunkd pid=13762732] splunkd --nodaemon -p 8089 _internal_exec_splunkd process-runner]
splunk 13762732 1900760 2 12:21:55 - 0:21 splunkd --nodaemon -p 8089 _internal_exec_splunkd
[root@bdaix71]:/root # lssrc -g splunk
Subsystem Group PID Status
splunkd splunk 13762732 active
Stop Splunk as root
using direct call to binary:
[root@bdaix71]:/root # /opt/splunk/uf/bin/splunk stop
Stopping splunkd...
Shutting down. Please wait, as this may take a few minutes.
0513-044 The splunkd Subsystem was requested to stop.
.
Stopping splunk helpers...
Done.
[root@bdaix71]:/root # ps -ef|grep [s]plunk
(no output)
[root@bdaix71]:/root # lssrc -g splunk
Subsystem Group PID Status
splunkd splunk inoperative
Login as the splunk
non-privileged user and attempt to start the daemon (this still fails):
[root@bdaix71]:/root # su - splunk
[splunk@bdaix71]:/opt/splunk $ /opt/splunk/uf/bin/splunk start
Splunk> CSI: Logfiles.
Checking prerequisites...
Checking mgmt port [8089]: open
Checking conf files for problems...
Done
All preliminary checks passed.
Starting splunk server daemon (splunkd)...
reading /etc/inittab: Permission denied
[splunk@bdaix71]:/opt/splunk $ ^D
[root@bdaix71]:/root # ps -ef|grep [s]plunk
(no output)
[root@bdaix71]:/root # lssrc -g splunk
Subsystem Group PID Status
splunkd splunk inoperative
Login as the splunk
non-privileged user and attempt to stop the daemon (this also fails). Note that the daemon has been started between this testcase and the previous one:
[root@bdaix71]:/root # su - splunk
[splunk@bdaix71]:/opt/splunk $ /opt/splunk/uf/bin/splunk status
splunkd is running (PID: 13762732).
splunk helpers are running (PIDs: 8781916).
[splunk@bdaix71]:/opt/splunk $ /opt/splunk/uf/bin/splunk stop
reading /etc/inittab: Permission denied
[splunk@bdaix71]:/opt/splunk $ ps -ef|grep [s]plunk
splunk 8781916 13762732 0 12:23:31 - 0:00 [splunkd pid=13762732] splunkd --nodaemon -p 8089 _internal_exec_splunkd process-runner]
splunk 13762732 1900760 3 12:21:55 - 0:22 splunkd --nodaemon -p 8089 _internal_exec_splunkd
Reboot server and verify Splunk is running (after reconnecting):
[root@bdaix71]:/root # ps -ef|grep [s]plunk
splunk 7078106 3735728 0 12:52:01 - 0:11 splunkd --nodaemon -p 8089 _internal_exec_splunkd
splunk 9699386 7078106 0 12:55:00 - 0:00 [splunkd pid=7078106] splunkd --nodaemon -p 8089 _internal_exec_splunkd process-runner]
[root@bdaix71]:/root # lssrc -g splunk
Subsystem Group PID Status
splunkd splunk 7078106 active
Has anyone found a resolution to this? We are encountering the same issue.,We are having this same issue. Has anyone come across a solution?