I am trying to UPGRADE using Ansible, I kick off the playbook via the bastion host. Here are the tasks.
1. copy the install file to remote
2. stop the Splunk service
3. install Splunk Forwarder 9.1
4. reboot
5. start the Splunk service
All are fine until step 4. I ssh to the specific host and checked the status. It was not running. I scratched my head and tried something like below.
sudo to root
/opt/splunkforwarder/bin/splunk version
it prompted for the license & perform upgrade message. I typed Y for both options.
after a few minutes (it showed a message to disable boot start), returned to prompt.
disabled boot start
reboot
sudo systemctl start splunk
Finally, it's up & running.
How do I fix step 5. I have 100s of ec2 instances to upgrade.
Hi
1st you should go throug 8.1 from 7.x to 9.0.1, you shouldn't go directly from 7.x to 9.x as it's not a supported upgrade step!
My own order to do it with ansible:
There is no need for reboot then node. And if you are rebooting it be sure that you enable boot-start before it. I cannot recall in which version there was some changes on systemd boot start option and for that reason (and some other too) it's good to re-enable it when you upgrade UF.
If you have some older versions like 7.x or even 6.x you must do this update in several steps. Just check that update path and do it from one supported versions at time like 7.x -> 8.0 or 8.1 (if source was 7.3). You must check those versions from Splunk's documentation.
r. Ismo
Hi
1st you should go throug 8.1 from 7.x to 9.0.1, you shouldn't go directly from 7.x to 9.x as it's not a supported upgrade step!
My own order to do it with ansible:
There is no need for reboot then node. And if you are rebooting it be sure that you enable boot-start before it. I cannot recall in which version there was some changes on systemd boot start option and for that reason (and some other too) it's good to re-enable it when you upgrade UF.
If you have some older versions like 7.x or even 6.x you must do this update in several steps. Just check that update path and do it from one supported versions at time like 7.x -> 8.0 or 8.1 (if source was 7.3). You must check those versions from Splunk's documentation.
r. Ismo
I tried all the steps. Now, something went wrong in step 9.
9. Start UF with systemctl as a service
It threw, failed to start splunk.service: Unit not found.
I ssh to the node and checked the status. It shows, Active: active (exited).
I tried to start, sudo systemctl start splunk
failed to start splunk.service: Unit not found.
I tried on another node. Splunk.service disappears after upgrade.
In your example you are using old init way not systemd. You should use
[sudo] $SPLUNK_HOME/bin/splunk enable boot-start -user bob -systemd-managed 1
I strongly recommended you to run UF as separate “splunk” -user not as a root which is security risk! With UF 9 there are some new features to do it. See
https://docs.splunk.com/Documentation/Forwarder/9.0.1/Forwarder/Installleastprivileged
https://docs.splunk.com/Documentation/Splunk/9.0.1/Admin/ConfigureSplunktostartatboottime
Finally, the below tasks seem to be working.
Copy the software
Stop Splunk
Upgrade UF
Disable boot-start
Start Splunk
Stop Splunk
Reboot
Check uptime
Start as service
Note: enable boot-start before re-boot was not needed as it was already enabled!
Thank you. I already tried with -systemd-managed 1
It still says unit not found. It's not able to find splunk.service.
sudo systemctl | grep splunk
splunk.service
Thanks for your response.
Regarding step 8, the Splunk user is the root. Is it sufficient to issue the below command?
8. Enable boot start as root with systemd + splunk user
/opt/splunkforwarder/bin/splunk enable boot-start