Getting Data In

[AIX] Universal Forwarder requires read access on `/etc/inittab` or else daemon won't start

dvanzuijlekom
Engager

When installing the Splunk 6.1.1 Universal Forwarder on AIX7.1, splunkd seems to require read access on /etc/inittab or else the daemon will fail to start. We run the Splunk daemon under a limited (non-root) useraccount and the inittab file is protected from unauthorized access.

Steps to reproduce:

[root@bdaix71]:/tmp # su - splunk
[splunk@bdaix71]:/opt/splunk $ uf/bin/splunk version
Splunk Universal Forwarder 6.1.1 (build 207789)
[splunk@bdaix71]:/opt/splunk $ uf/bin/splunk status
splunkd is not running.
[splunk@bdaix71]:/opt/splunk $ uf/bin/splunk start

Splunk> CSI: Logfiles.

Checking prerequisites...
        Checking mgmt port [8089]: open
        Checking conf files for problems...
        Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
reading /etc/inittab: Permission denied
[splunk@bdaix71]:/opt/splunk $ uf/bin/splunk status
splunkd is not running.

Permissions on /etc/inittab:

[root@bdaix71]:/tmp # ls -la /etc/inittab
-rw-------    1 root     system         3155 Feb 24 16:42 /etc/inittab

After changing /etc/inittab to group/worldreadable:

[root@bdaix71]:/tmp # chmod 644 /etc/inittab
[root@bdaix71]:/tmp # su - splunk
[splunk@bdaix71]:/opt/splunk $ uf/bin/splunk status
splunkd is not running.
[splunk@bdaix71]:/opt/splunk $ uf/bin/splunk start

Splunk> CSI: Logfiles.

Checking prerequisites...
        Checking mgmt port [8089]: open
        Checking conf files for problems...
        Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
Declared role=universal_forwarder.
Done

[splunk@bdaix71]:/opt/splunk $ uf/bin/splunk status
splunkd is running (PID: 7536766).
splunk helpers are running (PIDs: 5111876).

Starting splunk in --debug mode does not reveal extra information regarding the /etc/inittab file. Logging in var/log/splunk/ does not mention the inittab file and grepping for inittab only lists references in the splunkd binary.

Read access on this file seems pointless and is probably only required when the sysadmin wants to use the autostart capabilities(?).

0 Karma

antonyhan
Path Finder

we have the same problem with UF version 6.2.2-255606.
does anyone have a fix for it?

Thanks.

0 Karma

dvanzuijlekom
Engager

After simply ignoring this issue for quite a while, it has become relevant again, since we are currently in the process of upgrading from Splunk v5 to v6. One of my colleagues found a workaround which appears to partially solve our problems. I am posting this workaround for future reference and peer review.

1: Enable SRC subsystem
First, enable the Splunk SRC subsystem in AIX and instruct it to use the non-privileged useraccount (we'll use splunk, in this case):
/opt/splunk/uf/bin/splunk enable boot-start -user splunk

2: Remove SPLUNK_OS_USER limitation
Now, you need to remove the SPLUNK_OS_USER line in splunk-launch.conf. This will be added automatically by the previous command. If you don't, you won't be able to run the splunk start command as a user other than the one you specified earlier (and this will still fail due to the root cause of the inittab problem not being solved).

That's it. You will now be able to either use the default AIX startsrc -g splunk and stopsrc -g splunk commands and even run the binary as the root user. The daemon will drop privileges to the previously specified useraccount.

What you (still) won't be able to do
You still won't be able to run the splunk start and splunk stop commands directly as the non-privileged account. This may be a problem for some.

Testcases
After performing the actions in 1 and 2, the following testcases were carried out sequentially:

Starting point, Splunk is not running:

[root@bdaix71]:/root # oslevel -s
7100-03-01-1341
[root@bdaix71]:/root # /opt/splunk/uf/bin/splunk version
Splunk Universal Forwarder 6.2.1 (build 245427)
[root@bdaix71]:/root # ps -ef|grep [s]plunk
(no output)
[root@bdaix71]:/root # lssrc -g splunk
Subsystem         Group            PID          Status
 splunkd          splunk                        inoperative

Start Splunk as root using the AIX SRC subsystem:

[root@bdaix71]:/root # startsrc -g splunk
0513-059 The splunkd Subsystem has been started. Subsystem PID is 4849824.
[root@bdaix71]:/root # ps -ef|grep [s]plunk
  splunk  4849824  1900760   0 12:19:37      -  0:00 /opt/splunk/uf/bin/splunk _internal_exec_splunkd
  splunk 11141354  4849824 120 12:19:42      -  0:00 /opt/splunk/uf/bin/splunkd btool server list general --no-log
[root@bdaix71]:/root # lssrc -g splunk
Subsystem         Group            PID          Status
 splunkd          splunk           4849824      active

Stop Splunk as root using the AIX SRC subsystem:

[root@bdaix71]:/root # stopsrc -g splunk
0513-044 The splunkd Subsystem was requested to stop.
[root@bdaix71]:/root # lssrc -g splunk
Subsystem         Group            PID          Status
 splunkd          splunk                        inoperative
[root@bdaix71]:/root # ps -ef|grep [s]plunk
(no output)

Start Splunk as root using direct call to binary:

[root@bdaix71]:/root # /opt/splunk/uf/bin/splunk start

Splunk> CSI: Logfiles.

Checking prerequisites...
        Checking mgmt port [8089]: open
        Checking conf files for problems...
        Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
0513-059 The splunkd Subsystem has been started. Subsystem PID is 13762732.
Done

[root@bdaix71]:/root # ps -ef|grep [s]plunk
  splunk  8781916 13762732   0 12:23:31      -  0:00 [splunkd pid=13762732] splunkd --nodaemon -p 8089 _internal_exec_splunkd process-runner]
  splunk 13762732  1900760   2 12:21:55      -  0:21 splunkd --nodaemon -p 8089 _internal_exec_splunkd
[root@bdaix71]:/root # lssrc -g splunk
Subsystem         Group            PID          Status
 splunkd          splunk           13762732     active

Stop Splunk as root using direct call to binary:

[root@bdaix71]:/root # /opt/splunk/uf/bin/splunk stop
Stopping splunkd...
Shutting down.  Please wait, as this may take a few minutes.
0513-044 The splunkd Subsystem was requested to stop.
.
Stopping splunk helpers...

Done.
[root@bdaix71]:/root # ps -ef|grep [s]plunk
(no output)
[root@bdaix71]:/root # lssrc -g splunk
Subsystem         Group            PID          Status
 splunkd          splunk                        inoperative

Login as the splunk non-privileged user and attempt to start the daemon (this still fails):

[root@bdaix71]:/root # su - splunk
[splunk@bdaix71]:/opt/splunk $ /opt/splunk/uf/bin/splunk start

Splunk> CSI: Logfiles.

Checking prerequisites...
        Checking mgmt port [8089]: open
        Checking conf files for problems...
        Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
reading /etc/inittab: Permission denied
[splunk@bdaix71]:/opt/splunk $ ^D
[root@bdaix71]:/root # ps -ef|grep [s]plunk
(no output)
[root@bdaix71]:/root # lssrc -g splunk
Subsystem         Group            PID          Status
 splunkd          splunk                        inoperative

Login as the splunk non-privileged user and attempt to stop the daemon (this also fails). Note that the daemon has been started between this testcase and the previous one:

[root@bdaix71]:/root # su - splunk
[splunk@bdaix71]:/opt/splunk $ /opt/splunk/uf/bin/splunk status
splunkd is running (PID: 13762732).
splunk helpers are running (PIDs: 8781916).
[splunk@bdaix71]:/opt/splunk $ /opt/splunk/uf/bin/splunk stop
reading /etc/inittab: Permission denied
[splunk@bdaix71]:/opt/splunk $ ps -ef|grep [s]plunk
  splunk  8781916 13762732   0 12:23:31      -  0:00 [splunkd pid=13762732] splunkd --nodaemon -p 8089 _internal_exec_splunkd process-runner]
  splunk 13762732  1900760   3 12:21:55      -  0:22 splunkd --nodaemon -p 8089 _internal_exec_splunkd

Reboot server and verify Splunk is running (after reconnecting):

[root@bdaix71]:/root # ps -ef|grep [s]plunk
  splunk  7078106  3735728   0 12:52:01      -  0:11 splunkd --nodaemon -p 8089 _internal_exec_splunkd
  splunk  9699386  7078106   0 12:55:00      -  0:00 [splunkd pid=7078106] splunkd --nodaemon -p 8089 _internal_exec_splunkd process-runner]
[root@bdaix71]:/root # lssrc -g splunk
Subsystem         Group            PID          Status
 splunkd          splunk           7078106      active

dvietze
New Member

Has anyone found a resolution to this? We are encountering the same issue.,We are having this same issue. Has anyone come across a solution?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...