Change splunk to run as user 'splunk' now splunkweb fails to start with the error "Error starting splunkweb."

[stefanlasiewski]

This is a Scientific Linux 6.1 system (Equivalent to RHEL 6.1).

I installed Splunk as the root user, and it starts and runs fine as root.

Now I want to run Splunk as the user 'splunk' instead of as the user 'root'. So I follow the instructions at RunSplunkasadifferentornon-rootuser. That page says (in bold)

Then, before you start Splunk for the
first time, change the ownership of
the splunk directory to the desired
user.

But in this case, I have already installed Splunk as root and have already started Splunk as root.

# useradd splunk
# groupadd splunk
# chown -R splunk:splunk $SPLUNK_HOME
# ls -ld $SPLUNK_HOME
drwxr-xr-x 9 splunk splunk 4096 Dec  8 13:10 /data/splunk

And then I try to start splunk, as the user splunk. The following command is from /etc/init.d/splunk. Splunkd can start, but splunkweb fails to start.

# /bin/su splunk -c "/data/splunk/bin/splunk start"

Splunk> The IT Search Engine.

Checking prerequisites...
        Checking http port [8000]: open
        Checking mgmt port [8089]: open
        Checking configuration...  Done.
        Checking index directory...
        Validated databases: _audit _blocksignature _internal _perf_report _perf_test _thefishbucket history main splunkit_idxtest summary
        Done

Success
        Checking conf files for typos...
All preliminary checks passed.

Starting splunk server daemon (splunkd)... 
                                                           [  OK  ]
Error starting splunkweb.                                  [FAILED]
Done.Starting splunkweb... 

I looked in $SPLUNK_HOME/var/log/splunk/ (web_service.log splunkd.log) and I see absolutely no indication of failure. The splunkweb service failed without writing any information to those logs.

I ran an strace, but can't figure out what is failing:

# su - splunk -c "/usr/bin/strace /data/splunk/bin/splunk start splunkweb"
...
...
stat("/data/splunk/etc/auth/splunkweb", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
stat("/data/splunk/etc/auth/splunkweb/cert.pem", {st_mode=S_IFREG|0600, st_size=802, ...}) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7ffa303b39d0) = 5391
wait4(5391, [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], 0, NULL) = 5391
--- SIGCHLD (Child exited) @ 0 (0) ---
stat("/etc/rc.d/init.d/functions", {st_mode=S_IFREG|0644, st_size=17921, ...}) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7ffa303b39d0) = 5393
wait4(5393, Error starting splunkweb.                      [FAILED]
[{WIFEXITED(s) && WEXITSTATUS(s) == 1}], 0, NULL) = 5393
--- SIGCHLD (Child exited) @ 0 (0) ---
write(1, "Starting splunkweb... ", 22Starting splunkweb... )  = 22
exit_group(1)   

Why does splunkweb fail to start?

[theunf]

How about running as splunk user and then decide to run on tcp/443 ?

Stop splunk
Change ownership of files and folders to root.root
Start splunk again, right ?

After lots of permission erros splunkweb could not start.

Stop splunk
Change ownership to splunk.splunk
Start splunk AS root

Working ... but the onwership of changed files became a mess 😉

That´s for the tgz version of 6.1.x on Ubuntu or RHES.

[stefanlasiewski]

For our needs, I put Apache in front of Splunk Web and I redirect the traffic using a reverse proxy. This gives me the power, flexibility and security of Apache.

However, it's also possible to use iptables to redirect port 443 to a high-numbered port that Splunk can use. The following answer talks about port 514/UDP but the similar rules can apply to port 443/TCP: http://answers.splunk.com/answers/63898/how-to-listen-to-port-udp-514-when-splunk-is-not-root

[stefanlasiewski]

@theunf : You don't need to chown the files to root:root in order to listen on port 443. You only need to start Splunk as root. But as you say, this does leave a mess in your $SPLUNK_HOME, as some files are owned by Splunk while others are owned as user Root.

[stefanlasiewski]

We got this fixed eventually. See my answer below.

[stefanlasiewski · Accepted solution]

Took a while, but we finally found the cause of my problem with the help of Splunk Technical Support and many hours on the phone.

Splunk did not have permission to read /etc/httpd/mime.types , which is a special file that we created for our own local Apache installation. I don't know why Splunk is trying to read that file or why the error is not properly reported to the logfile. According to Splunk support, this is a bug and will get fixed.

I changed the permissions on that file so that the user splunk could read that file. Splunk is now able to be run as user splunk.

[stefanlasiewski]

Splunk support asked me to reinstall Splunk, but this problem still happens.

[stefanlasiewski]

I had a long, long discussion with Splunk support about this, and we never resolved this. In addition, Splunk never tells me why this fails, even with Debug logging cranked up.

[Brian_Osburn]

I don't think you really need to do anything to run splunk as a user splunk - matter of fact when you install it, it allows you to run as splunk.

The next question is have you changed what port splunk is listening on?

[Brian_Osburn]

hrrm..the only thing i could think of is if you had tried to chown the directory while not as root. If you still can't get it started, I'd suggest opening a support case..

[stefanlasiewski]

Splunkweb is listening on port 8000. As the user splunk, I ran nc -l 8000 (Netcat) to confirm that the user splunk can listen on that port.