Hi There,
We've a standalone Splunk instance v8.2.2.1 deployed on a RHEL server which is EOL; we wish to migrate to a newer OS Amazon Linux (AL) 2023 OS-- rather than performing an in-place upgrade. Instead of using the most recent version of Splunk enterprise, we still wish to adopt a more conservative approach and choose 9.0.x (we've UFs that are older version 7.x and 8.x). Please let me know where can i download 9.0.x version of Splunk enterprise as it's not here:
https://www.splunk.com/en_us/download/previous-releases.html
Thanks!
Hi All:
Thank you and I appreciate your response.
We have a standalone instance of Splunk indexer and I double-checked and for the most part we're using 8.2.9 version of SplunkUF.
Additionally, since Splunk Enterprise 9.2.7 is the version in the 9.x.x family that supports Amazon Linux, we'll go for the same version.
Currently the indexes' physical location is spread across volumes that are mounted on the Splunk indexer host at OS level.
Please let us know if this is the right approach and if there are any stages we're missing in regards to the data cutover from the old to the new server.
• Install Splunk Enterprise 9.2.7 on a new AL2023 server
• Take a snapshot of the old server's volumes that contains indexed data, then connect them to the new one using the same mount point.
• Copy the entire $SPLUNK_HOME/etc directory from the old server to the new server
• Copy indexed data from $SPLUNK_DB (/opt/splunk/var/lib/splunk) to the new server
• Detach & attach publicIP/EIP from old to the new server
I don’t believe that you have any issues with those 8.x.x UFs with splunk 9.3.x or even. 9.4.x. Those will work together, maybe some modifications are needed, but probably none.
Here is one old post which points to some other post based on your environment.
If/when you could do a new host which you can use for some testing this shouldn’t be an issue. Just test it with test systems with instructions from those above posts. When you have check and approved those test then just do real migration.
I’m not 100% sure that there is not any issues with amz2023 version. I have some feelings that there could be something which need to configure separately e.g. cgroups or something else? You probably find more details from https://splunkcommunity.slack.com/archives/C03M9ENE6AD
Yes. The range of interoperability between UFs and receiving components (intermediate forwarders/indexers) is quite big. Even if the official documentation doesn't list something as supported, things might just work. I've had UFs as old as 6.6 sending to version 9 indexers and it ran OK. There might be a minor issue with v9 UFs sending to older indexers because new UFs generate config change events which are supposed to go to indexes not present on older Splunk instances. The temporary walkaround for this is to disable the config tracker inputs on the UFs until the indexers are upgraded to v9. But even if you don't do that, they will generally work, it's just that those events will either land in your last chance index or will generate a warning about non-existent index and get dropped completely.
The general idea is OK but there are details which can pop up unexpectedly here and there.
1. I assume (never used it myself) that Amazon Linux is also an RPM-based distro and you'll be installing Splunk the same way it was installed before.
2. Remember to shut down Splunk service before moving the data. And of course don't start the new instance before you copy the data.
3. I'm not sure why you want to snapshot the volumes. For backup in case you need to roll back?
4. You might have other dependencies lying around, not included in $SPLUNK_HOME - for example certificates.
5. If you move whole filesystems between server instances the UIDs and GIDs might not match and you might need to fix your accesses.
Oh, and most importantly - I didn't notice that at first - DON'T UPGRADE AND MOVE AT THE SAME TIME! Either upgrade and then do the move to the same version on a new server or move to the same 8.x you have now and then upgrade on the new server.
Hi @PickleRick,
Thank you so much for your help...Please find the comments inline:
1. I assume (never used it myself) that Amazon Linux is also an RPM-based distro and you'll be installing Splunk the same way it was installed before.
Yes, Amazon Linux natively supports RPM package installer
2. Remember to shut down Splunk service before moving the data. And of course don't start the new instance before you copy the data.
Got it.
3. I'm not sure why you want to snapshot the volumes. For backup in case you need to roll back?
Yes, correct..in case there is a need to rollback
4. You might have other dependencies lying around, not included in $SPLUNK_HOME - for example certificates.
In our case, the ssl certificates are deployed under /opt/splunk/etc/certs/ as the ssl offloading is directly on the server and there is no loadbalancer or proxy in the front. Can you think of anything else that may deployed outside of /opt/splunk
5. If you move whole filesystems between server instances the UIDs and GIDs might not match and you might need to fix your accesses.
Can we recursively chown the files on the new server after migration to ensure correct ownership, hope that should take care of it
sudo chown -R splunk:splunk /opt/splunk
Oh, and most importantly - I didn't notice that at first - DON'T UPGRADE AND MOVE AT THE SAME TIME! Either upgrade and then do the move to the same version on a new server or move to the same 8.x you have now and then upgrade on the new server.
Sure I prefer doing the latter, but the older version of Splunk Enterprise 8.2.2.1 does not support Amazon Linux.
4. That was the most obvious example. There might be some other dependencies - for example, if you're using dbconnect, you require JRE.
5. Yes, chowning should take care of it. But as I understood from your earlier comments, you have your index volume(s) outside /opt/splunk. You need to take care of its ownership as well.
Hi @PickleRick
I did the following in my test environment and migration is successful. PLease let me know your thoughts on this procedure
* Installed Splunk 9.2.7 on a fresh AL2023 server, verified that the web console was accessible, and confirmed I could log in.
* Stopped the Splunk service
* Copied the /opt/splunk/etc/ and /opt/splunk/var/lib/splunk directories from the 8.2.2.1 server to the new server
* Mounted the necessary volumes from the old server to the new one, ensuring the index data was available
* Uninstalled Splunk 9.2.7
* Noted that, after uninstalling, the etc and db directories remained intact.
* Reinstalled Splunk 9.2.7
* During the initial start with sudo /opt/splunk/bin/splunk start --accept-license, I observed that Splunk successfully migrated the configuration and settings in etc to be compatible with 9.2.7.
* I now have a fully functional Splunk v9.2.7 instance with all historical indexed data present
As I said before - move an upgrade in one move was risky.
I don't see the point in uninstalling and reinstalling the package.
Hi @PickleRick
Agreed, however, when i start the Splunk after accepting the license agreement, i run into the following screenshot which takes care of the seamless migration, I believe what I'm doing must be a documented procedure and nothing unusual and it also creates a migration logs with the details of what was done during the process... please lemme know your thoughts!!
Thanks for your help & Happy 4th!!
Download migration log from here:
https://limewire.com/d/Jd4GD#NEdMoeWwVg
There are several things which "work" but which are unsupported and might bite you here and there at some point. Just saying. As @isoutamo pointed out (I admit I didn't bother to check this one), straight jump to 9.2 from your old version isn't supported so whilie the migration seems to have gone well, you might have skipped some step normally performed on migration to 9.0, for example, which later versions might rely on.
Again - you might get away with doing unsupported things if you're lucky. But you might not. And debugging will be more difficult later if you have some issues lingering from a few versions back.
I can share the log file from the migration, however, i don't see an option here to upload. PLease do you know if there a way to share/upload log files.
Thanks
I agree with @PickleRick, don’t move and upgrade at same time! Also you shouldn’t upgrade directly from 8.2.x to 9.2.x. Only supported way is migrate over one version like 8.2-> 9.0 -> 9.2 etc. and you must start your node(s) after upgrade to each separate versions.
Splunk doesn’t support rollback of version upgrade. So uninstall version is not needed/suggested.
Also you should check in Amz23 at least systemd startup settings as those are somehow different than in RHEL. Cgroups default is v2 which needs some parameter changes etc. Also if your environment needs IMDS its version has changed to v2. Probably doesn’t affect to you unless yo are using some old AWS ta?
Hi @PickleRick @isoutamo
Thanks for taking time to reply. Much appreciated. I'd prefer to take the upgrade path of 8.2->9.0.x -> 9.2.7; unfortunately, 9.0 does not support AL2023.
Please take a moment to review the migration log file as didn't see any alarming (you may download from the link below), i did disable THP and verified the ulimits to match the recommended settings.
https://limewire.com/d/PDWiS#NfyxSpwkrX
Also, log ingestion is working properly . It's been a week since I upgraded our test instance of Splunk to 9.27 and there has been no issues. The health stats from Settings->Monitoring Console reports no issues. And we dont use workload management feature in Splunk from a cgroup compatibility standpoint we are covered...
With regards to SystemD, it was setup using the below commands in AL2023:
sudo /opt/splunk/bin/splunk enable boot-start -user splunk -systemd-managed 1
sudo systemctl daemon-reload
sudo systemctl start splunkd
sudo systemctl enable splunkd
As I said before - if it worked, good for you.
And I'm not gonna go through your logs. For several reasons.
Don't take it personally but this is a public forum where people give time whenever they want, the way they want. Answers is not a free support/consulting service. If you want something specific done - reach out to your local Splunk Partner or hire Professional Services. To put it bluntly - it costs money.
Secondly, you've already done the migration and apparently you're running the new system as prod. My looking into your logs won't change anything.
Thirdly, I might miss something, especially considering the first point.
You asked, you have been advised, you did otherwise. I won't pat you now on the back and say "it's ok". It might. But don't have to. And I don't have spare time just to ease your mind, to be frank.
I hope it is ok. But I won't check it.
You are going to have to contact Splunk Support for any older versions not on their website. I apologize for that inconvenience.
It is your environment and you need to do what you and your management team feel are the best things, but as a person employed in the Cyber Security arena, I feel that I should at least mention the following. None of this applies to your wanting to run 9.0.x It was the Splunk 7 and Splunk 8 that raised my antennae.
Actually, compression has been around for quite a long time and 7.x forwarders should support it.
Also, your protocol levels are way off.
Not to mention the bogus requirement to use 9.0+ DS to manage 7/8 version UFs.
Please refrain from posting AI-generated content.
And why would you go for 9.0 which is out of support? I'd strongly advise against that. Unless you have a very good reason for doing so (and a very very good support contract, other than us, mere mortals) it's unwise to keep your environment at an unsupported version (which applies to the current 8.2 as well).
Contact Splunk Support for versions not available on the web site.