Splunk indexer service: Why error "RHEL 7.1 systemd[1]: Failed to start SYSV: Splunk indexer service"?

[lraynal]

My Splunk indexer is not starting as a service on RHEL 7.1 on a fresh install.
It's starting ok as splunk user though.

 [root@myindexer ~]# systemctl status splunk
    splunk.service - SYSV: Splunk indexer service
       Loaded: loaded (/etc/rc.d/init.d/splunk)
       Active: failed (Result: exit-code) since mer. 2015-09-30 18:21:15 CEST; 4min 13s ago

    sept. 30 18:21:15 myindexer splunk[2938]: Starting Splunk...
    sept. 30 18:21:15 myindexer splunk[2938]: Splunk> Needle. Haystack. Found.
    sept. 30 18:21:15 myindexer splunk[2938]: Checking prerequisites...
    sept. 30 18:21:15 myindexer splunk[2938]: Checking http port [443]: already bound
    sept. 30 18:21:15 myindexer splunk[2938]: ERROR: The http port [443] is already bound.  Splunk needs to use this port.
    sept. 30 18:21:15 myindexer splunk[2938]: Would you like to change ports? [y/n]:
    sept. 30 18:21:15 myindexer splunk[2938]: Exiting due to --no-prompt.
    sept. 30 18:21:15 myindexer systemd[1]: splunk.service: control process exited, code=exited status=1
    sept. 30 18:21:15 myindexer systemd[1]: Failed to start SYSV: Splunk indexer service.
    sept. 30 18:21:15 myindexer systemd[1]: Unit splunk.service entered failed state.

Previously I did change Splunk Web server port to HTTPS / 443

    # echo "/opt/splunk/lib" > /etc/ld.so.conf.d/splunk.x86_64.conf
    # ldconfig
    # setcap 'cap_net_bind_service=+ep' /opt/splunk/bin/splunk
    # setcap 'cap_net_bind_service=+ep' /opt/splunk/bin/splunkd
    # su - splunk
    $ splunk --accept-license edit user admin -password $SPLUNK_PASSWORD -auth admin:changeme
    $ splunk set web-port 443
[..]

[aasraoui]

Hi,

would like to know where i can modify splunk db variable to point to a new directory with larger storage capacity.  

thanks

abdelillah

[gbedsaul1]

I'm getting a similar error to this, but I have no idea where it might be:

"""
[root@forwarder /opt/splunk]# systemctl -l status splunk
● splunk.service
Loaded: not-found (Reason: No such file or directory)
Active: failed (Result: exit-code) since Wed 2019-09-04 06:48:01 UTC; 49min ago

Sep 04 06:48:01 myforwarder splunk[4819]: and do not create a new session
Sep 04 06:48:01 myforwarder splunk[4819]: -f, --fast pass -f to the shell (for csh or tcsh)
Sep 04 06:48:01 myforwarder splunk[4819]: -s, --shell run shell if /etc/shells allows it
Sep 04 06:48:01 myforwarder splunk[4819]: -h, --help display this help and exit
Sep 04 06:48:01 myforwarder splunk[4819]: -V, --version output version information and exit
Sep 04 06:48:01 myforwarder splunk[4819]: For more details see su(1).
Sep 04 06:48:01 myforwarder systemd[1]: splunk.service: control process exited, code=exited status=1
Sep 04 06:48:01 myforwarder systemd[1]: Failed to start SYSV: Splunk indexer service.
Sep 04 06:48:01 myforwarder systemd[1]: Unit splunk.service entered failed state.
Sep 04 06:48:01 myforwarder systemd[1]: splunk.service failed.
"""

Especially since it's supposed to be running as a forwarder... Oy

[buntel]

I did the following and it worked. Don't ask me why since I am not an expert 😄
sudo chown -R splunk:splunk /opt/splunk

[DalJeanis]

The why on this is that you gave the splunk userid the ownership of all files in the /opt/splunk directory, and recursively (-R) below that. So that error was a file permissions issue for you.

[lraynal · Accepted solution]

This is in fact a problem with /opt/splunk/bin/splunk enable boot-start -user splunk
which installs a /etc/init.d/splunk that does everything as root, not splunk.

I added su splunk -c everywhere it's launching splunk, as in
su splunk -c "/opt/splunk/bin/splunk start --no-prompt --answer-yes"

[woodcock]

Google "splunk user bob docs". It is a sad situation that Splunk the enable boot-start command does not have an option for this.