Splunkd services are not starting on boot-start

[man03359]

Hi All,

Hope this message finds you well.

I have installed splunk on-prem on a linux box as a splunk user and have given proper permissions.

The azure VM gets shutsdown automatically at around 11 pm everyday and there is no auto start. For time being we are manually starting the VM.

My problem here is while installing the splunk instance, I have run the command enable boot-start and it was successful but the splunkd services does not start on its own. 

Can anyone please suggest what can be done to fix it?

Thanks in advance 🙂

[TheLawsOfChaos]

So there are a lot of questions to ask, as you state just linux. Is it debian or centos/redhat based? If it's redhat, are you using systemd?

https://docs.splunk.com/Documentation/Splunk/9.2.2/Admin/ConfigureSplunktostartatboottime

When you run 

[sudo] $SPLUNK_HOME/bin/splunk enable boot-start -user splunk

what sort of output do you get?

Keep in mind if you are using systemd there is an entire section in the documentation that goes over fighting that lovely beast. 

Have you checked /opt/splunk/var/log/splunk/splunkd.log to see if there are any issues with it attempting to autostart? Sometimes things such as permissions issues can also affect it. Are you able to manually start splunk as the splunk user and it boots up fine?

[man03359]

@TheLawsOfChaos ,

The Linux is a redhad.

And I have already created  a user called splunk, so under this path -

cd /opt/splunk/bin/

I am running this command -

sudo ./splunk enable boot-start.

I am able to manually start the services using-

sudo ./splunk start

[TheLawsOfChaos]

So when running the Splunk service, you do not want to be running it as root (which is primarily what sudo does). Since you have run some of the commands via sudo, that means some of the file permissions most likely were changed to root owning it. 

You would want to follow these steps:

First, you need to ensure that the splunk user/group owns the files, since you have been running it as root (sudo)

1) 

sudo chown -R splunk:splunk /opt/splunk

Second, you want to become the splunk user

2) 

sudo su splunk

Then you want to run your commands as normal

3)

./splunk enable boot-start -user splunk

or 

./splunk enable boot-start -user splunk -systemd-managed 1

if you are using systems on your system.

By running the commands as the splunk user, you ensure that the splunk user maintains ownership over /opt/splunk, and that means that the enable boot start will be able to work. I think if you checked your linux logs, you would see during boot up there are probably permission errors stating that the user splunk does not have access to the /opt/splunk folder, due to the sudo issues.

After doing this, while still as the splunk user you can run ./splunk start.

If you don't want to do sudo su splunk, to become the user you can use something like this instead:

sudo -H -u splunk $SPLUNK_HOME/bin/splunk start

This will let you use sudo as your user, tell it to act as the splunk user, and then start splunk. This method of sudo usage could replace directly sudo su splunk if needed.