topic Re: Splunk Forwarder Agent from 7.x to 9.1 on RHEL 7.9- How do I fix step 5? in Splunk Enterprise

Splunk Forwarder Agent from 7.x to 9.1 on RHEL 7.9- How do I fix step 5?

skpdevops — Mon, 17 Oct 2022 14:35:10 GMT

I am trying to UPGRADE using Ansible, I kick off the playbook via the bastion host. Here are the tasks.

1. copy the install file to remote

2. stop the Splunk service

3. install Splunk Forwarder 9.1

4. reboot

5. start the Splunk service

All are fine until step 4. I ssh to the specific host and checked the status. It was not running. I scratched my head and tried something like below.

sudo to root

/opt/splunkforwarder/bin/splunk version

it prompted for the license & perform upgrade message. I typed Y for both options.

after a few minutes (it showed a message to disable boot start), returned to prompt.

disabled boot start

reboot

sudo systemctl start splunk

Finally, it's up & running.

How do I fix step 5. I have 100s of ec2 instances to upgrade.

Re: Splunk Forwarder Agent from 7.x to 9.1 on RHEL 7.9- How do I fix step 5?

isoutamo — Mon, 17 Oct 2022 15:21:52 GMT

1st you should go throug 8.1 from 7.x to 9.0.1, you shouldn't go directly from 7.x to 9.x as it's not a supported upgrade step!

My own order to do it with ansible:

Check version if this is needed or not
Copy uf RPM/tar package to node if needed
Stop UF (as splunk UF user)
Disable boot start (as root)
Update to supported version
Start UF as splunk user with --accept-license --answer-yes
Stop UF as splunk user
Enable boot start as root with systemd + splunk user
Start UF with systemctl as a service

There is no need for reboot then node. And if you are rebooting it be sure that you enable boot-start before it. I cannot recall in which version there was some changes on systemd boot start option and for that reason (and some other too) it's good to re-enable it when you upgrade UF.

If you have some older versions like 7.x or even 6.x you must do this update in several steps. Just check that update path and do it from one supported versions at time like 7.x -> 8.0 or 8.1 (if source was 7.3). You must check those versions from Splunk's documentation.

r. Ismo

Re: Splunk Forwarder Agent from 7.x to 9.1 on RHEL 7.9- How do I fix step 5?

skpdevops — Mon, 17 Oct 2022 16:35:13 GMT

Thanks for your response.

Regarding step 8, the Splunk user is the root. Is it sufficient to issue the below command?

8. Enable boot start as root with systemd + splunk user

/opt/splunkforwarder/bin/splunk enable boot-start

Re: Splunk Forwarder Agent from 7.x to 9.1 on RHEL 7.9- How do I fix step 5?

skpdevops — Mon, 17 Oct 2022 20:37:54 GMT

I tried all the steps. Now, something went wrong in step 9.

9. Start UF with systemctl as a service

It threw, failed to start splunk.service: Unit not found.

I ssh to the node and checked the status. It shows, Active: active (exited).

I tried to start, sudo systemctl start splunk

failed to start splunk.service: Unit not found.

I tried on another node. Splunk.service disappears after upgrade.

Re: Splunk Forwarder Agent from 7.x to 9.1 on RHEL 7.9- How do I fix step 5?

isoutamo — Tue, 18 Oct 2022 05:19:25 GMT

In your example you are using old init way not systemd. You should use

[sudo] $SPLUNK_HOME/bin/splunk enable boot-start -user bob -systemd-managed 1

I strongly recommended you to run UF as separate “splunk” -user not as a root which is security risk! With UF 9 there are some new features to do it. See

https://docs.splunk.com/Documentation/Forwarder/9.0.1/Forwarder/Installleastprivileged

https://docs.splunk.com/Documentation/Splunk/9.0.1/Admin/ConfigureSplunktostartatboottime

Re: Splunk Forwarder Agent from 7.x to 9.1 on RHEL 7.9- How do I fix step 5?

skpdevops — Tue, 18 Oct 2022 12:11:43 GMT

Thank you. I already tried with -systemd-managed 1

It still says unit not found. It's not able to find splunk.service.

Re: Splunk Forwarder Agent from 7.x to 9.1 on RHEL 7.9- How do I fix step 5?

skpdevops — Wed, 19 Oct 2022 14:03:42 GMT

Finally, the below tasks seem to be working.
Copy the software
Stop Splunk
Upgrade UF
Disable boot-start
Start Splunk
Stop Splunk
Reboot
Check uptime
Start as service

Note: enable boot-start before re-boot was not needed as it was already enabled!

Re: Splunk Forwarder Agent from 7.x to 9.1 on RHEL 7.9- How do I fix step 5?

isoutamo — Wed, 19 Oct 2022 16:12:01 GMT

Is th name of service splunkd (or was it Splunkd) instead of splunk?

Re: Splunk Forwarder Agent from 7.x to 9.1 on RHEL 7.9- How do I fix step 5?

skpdevops — Wed, 19 Oct 2022 17:21:15 GMT

sudo systemctl | grep splunk

splunk.service