Splunk Forwarder Agent from 7.x to 9.1 on RHEL 7.9- How do I fix step 5?

[skpdevops]

I am trying to UPGRADE using Ansible, I kick off the playbook via the bastion host. Here are the tasks.

1. copy the install file to remote

2. stop the Splunk service

3. install Splunk Forwarder 9.1

4. reboot

5. start the Splunk service

All are fine until step 4. I ssh to the specific host and checked the status. It was not running. I scratched my head and tried something like below.

sudo to root

/opt/splunkforwarder/bin/splunk version

it prompted for the license & perform upgrade message. I typed Y for both options.

after a few minutes (it showed a message to disable boot start), returned to prompt.

disabled boot start

reboot

sudo systemctl start splunk

Finally, it's up & running. 

How do I fix step 5. I have 100s of ec2 instances to upgrade.

[isoutamo · Accepted solution]

Hi

1st you should go throug 8.1 from 7.x to 9.0.1, you shouldn't go directly from 7.x to 9.x as it's not a supported upgrade step!

My own order to do it with ansible:

1. Check version if this is needed or not
2. Copy uf RPM/tar package to node if needed
3. Stop UF (as splunk UF user)
4. Disable boot start (as root)
5. Update to supported version
6. Start UF as splunk user with --accept-license --answer-yes
7. Stop UF as splunk user
8. Enable boot start as root with systemd + splunk user
9. Start UF with systemctl as a service

There is no need for reboot then node. And if you are rebooting it be sure that you enable boot-start before it. I cannot recall in which version there was some changes on systemd boot start option and for that reason (and some other too) it's good to re-enable it when you upgrade UF.

If you have some older versions like 7.x or even 6.x you must do this update in several steps. Just check that update path and do it from one supported versions at time like 7.x -> 8.0 or 8.1 (if source was 7.3). You must check those versions from Splunk's documentation.

r. Ismo

[skpdevops]

I tried all the steps. Now, something went wrong in step 9.

9. Start UF with systemctl as a service

It threw, failed to start splunk.service: Unit not found.

I ssh to the node and checked the status. It shows, Active: active (exited).

I tried to start, sudo systemctl start splunk

failed to start splunk.service: Unit not found.

I tried on another node. Splunk.service disappears after upgrade.

[isoutamo]

In your example you are using old init way not systemd. You should use 

[sudo] $SPLUNK_HOME/bin/splunk enable boot-start -user bob -systemd-managed 1

I strongly recommended you to run UF as separate “splunk” -user not as a root which is security risk! With UF 9 there are some new features to do it. See 

https://docs.splunk.com/Documentation/Forwarder/9.0.1/Forwarder/Installleastprivileged

https://docs.splunk.com/Documentation/Splunk/9.0.1/Admin/ConfigureSplunktostartatboottime

[skpdevops]

Finally, the below tasks seem to be working.
Copy the software
Stop Splunk
Upgrade UF
Disable boot-start
Start Splunk
Stop Splunk
Reboot
Check uptime
Start as service

Note: enable boot-start before re-boot was not needed as it was already enabled!

[skpdevops]

Thank you. I already tried with -systemd-managed 1

It still says unit not found. It's not able to find splunk.service.

[isoutamo]

Is th name of service splunkd (or was it Splunkd) instead of splunk?

[skpdevops]

sudo systemctl | grep splunk

splunk.service

[skpdevops]

Thanks for your response.

Regarding step 8, the Splunk user is the root. Is it sufficient to issue the below command?

8. Enable boot start as root with systemd + splunk user

/opt/splunkforwarder/bin/splunk enable boot-start