Splunk Migration from RHEL to AL2023

[venksel]

Hi There,

We've a standalone Splunk instance v8.2.2.1 deployed on a  RHEL server which is EOL; we wish to migrate to a newer OS Amazon Linux (AL) 2023 OS-- rather than  performing an in-place upgrade. Instead of using the most recent version of Splunk enterprise, we still wish to adopt a more conservative approach and choose 9.0.x (we've UFs that are older version 7.x and 8.x). Please let me know where can i download 9.0.x version of Splunk enterprise as it's not here:

https://www.splunk.com/en_us/download/previous-releases.html

Thanks!

[isoutamo]

Here is link to repository which you could use to download older splunk versions https://github.com/ryanadler/downloadSplunk/tree/main

[venksel]

Hi All:

Thank you and I appreciate your response.

We have a standalone instance of Splunk indexer and I double-checked and for the most part we're using 8.2.9 version of SplunkUF.

Additionally, since Splunk Enterprise 9.2.7 is the version in the 9.x.x family that supports Amazon Linux, we'll go for the same version.

Currently the indexes' physical location is spread across volumes that are mounted on the Splunk indexer host at OS level.

Please let us know if this is the right approach and if there are any stages we're missing in regards to the data cutover from the old to the new server.

• Install Splunk Enterprise 9.2.7 on a new AL2023 server
• Take a snapshot of the old server's volumes that contains indexed data, then connect them to the new one using the same mount point.
• Copy the entire $SPLUNK_HOME/etc directory from the old server to the new server
• Copy indexed data from $SPLUNK_DB (/opt/splunk/var/lib/splunk) to the new server
• Detach & attach publicIP/EIP from old to the new server

[isoutamo]

I don’t believe that you have any issues with those 8.x.x UFs with splunk 9.3.x or even. 9.4.x. Those will work together, maybe some modifications are needed, but probably none.

Here is one old post which points to some other post based on your environment.  

https://community.splunk.com/t5/Deployment-Architecture/Splunk-Migration-from-existing-server-to-a-n...

If/when you could do a new host which you can use for some testing this shouldn’t be an issue. Just test it with test systems with instructions from those above posts. When you have check and approved those test then just do real migration.

I’m not 100% sure that there is not any issues with amz2023 version. I have some feelings that there could be something which need to configure separately e.g. cgroups or something else? You probably find more details from https://splunkcommunity.slack.com/archives/C03M9ENE6AD

[PickleRick]

Yes. The range of interoperability between UFs and receiving components (intermediate forwarders/indexers) is quite big. Even if the official documentation doesn't list something as supported, things might just work. I've had UFs as old as 6.6 sending to version 9 indexers and it ran OK. There might be a minor issue with v9 UFs sending to older indexers because new UFs generate config change events which are supposed to go to indexes not present on older Splunk instances. The temporary walkaround for this is to disable the config tracker inputs on the UFs until the indexers are upgraded to v9. But even if you don't do that, they will generally work, it's just that those events will either land in your last chance index or will generate a warning about non-existent index and get dropped completely.

[isoutamo]

Exactly that way. And Splunk has changed requirements to have higher version in Indexers with version 9.x. (not sure which minor x was). Now you can officially have newer UF version than receiving HF/IDX version.

[PickleRick]

The general idea is OK but there are details which can pop up unexpectedly here and there.

1. I assume (never used it myself) that Amazon Linux is also an RPM-based distro and you'll be installing Splunk the same way it was installed before.

2. Remember to shut down Splunk service before moving the data. And of course don't start the new instance before you copy the data.

3. I'm not sure why you want to snapshot the volumes. For backup in case you need to roll back?

4. You might have other dependencies lying around, not included in $SPLUNK_HOME - for example certificates.

5. If you move whole filesystems between server instances the UIDs and GIDs might not match and you might need to fix your accesses.

Oh, and most importantly - I didn't notice that at first - DON'T UPGRADE AND MOVE AT THE SAME TIME! Either upgrade and then do the move to the same version on a new server or move to the same 8.x you have now and then upgrade on the new server.

[venksel]

Hi @PickleRick,

Thank you so much for your help...Please find the comments inline:

1. I assume (never used it myself) that Amazon Linux is also an RPM-based distro and you'll be installing Splunk the same way it was installed before.

Yes, Amazon Linux natively supports RPM package installer

2. Remember to shut down Splunk service before moving the data. And of course don't start the new instance before you copy the data.

Got it.

3. I'm not sure why you want to snapshot the volumes. For backup in case you need to roll back?

Yes, correct..in case there is a need to rollback

4. You might have other dependencies lying around, not included in $SPLUNK_HOME - for example certificates.

In our case, the ssl certificates are deployed under /opt/splunk/etc/certs/ as the ssl offloading is directly on the server and there is no loadbalancer or proxy in the front.  Can you think of anything else that may  deployed outside of /opt/splunk

5. If you move whole filesystems between server instances the UIDs and GIDs might not match and you might need to fix your accesses.

Can we recursively chown the files on the new server after migration to ensure correct ownership, hope that should take care of it

sudo chown -R splunk:splunk /opt/splunk

Oh, and most importantly - I didn't notice that at first - DON'T UPGRADE AND MOVE AT THE SAME TIME! Either upgrade and then do the move to the same version on a new server or move to the same 8.x you have now and then upgrade on the new server.

Sure I prefer doing the latter, but the older version of Splunk Enterprise 8.2.2.1 does not support Amazon Linux.

[PickleRick]

4. That was the most obvious example. There might be some other dependencies - for example, if you're using dbconnect, you require JRE.

5. Yes, chowning should take care of it. But as I understood from your earlier comments, you have your index volume(s) outside /opt/splunk. You need to take care of its ownership as well.

[venksel]

Hi @PickleRick 

I did the following in my test environment and migration is successful. PLease let me know your thoughts on this procedure

* Installed Splunk 9.2.7 on a fresh AL2023 server, verified that the web console was accessible, and confirmed I could log in.
* Stopped the Splunk service
* Copied the /opt/splunk/etc/ and /opt/splunk/var/lib/splunk directories from the 8.2.2.1 server to the new server
* Mounted the necessary volumes from the old server to the new one, ensuring the index data was available
* Uninstalled Splunk 9.2.7
* Noted that, after uninstalling, the etc and db directories remained intact.
* Reinstalled Splunk 9.2.7
* During the initial start with sudo /opt/splunk/bin/splunk start --accept-license, I observed that Splunk successfully migrated the configuration and settings in etc to be compatible with 9.2.7.
* I now have a fully functional Splunk v9.2.7 instance with all historical indexed data present

[PickleRick]

As I said before - move an upgrade in one move was risky.

I don't see the point in uninstalling and reinstalling the package.

[venksel1]

Hi @PickleRick 

Agreed, however, when i start the Splunk after accepting the license agreement, i run into the following screenshot which takes care of the seamless migration, I believe what I'm doing must be a documented procedure and nothing unusual and it also creates a migration logs with the details of what was done during the process... please lemme know your thoughts!!

Thanks for your help & Happy 4th!!

Download migration log from here:

https://limewire.com/d/Jd4GD#NEdMoeWwVg

[PickleRick]

There are several things which "work" but which are unsupported and might bite you here and there at some point. Just saying. As @isoutamo pointed out (I admit I didn't bother to check this one), straight jump to 9.2 from your old version isn't supported so whilie the migration seems to have gone well, you might have skipped some step normally performed on migration to 9.0, for example, which later versions might rely on.

Again - you might get away with doing unsupported things if you're lucky. But you might not. And debugging will be more difficult later if you have some issues lingering from a few versions back.

[venksel]

I can share the log file from the migration, however, i don't see an option here to upload. PLease do you know if there a  way to share/upload log files.

Thanks

[isoutamo]

I agree with @PickleRick, don’t move and upgrade at same time! Also you shouldn’t upgrade directly from 8.2.x to 9.2.x. Only supported way is migrate over one version like 8.2-> 9.0 -> 9.2 etc. and you must start your node(s) after upgrade to each separate versions.

Splunk doesn’t support rollback of version upgrade. So uninstall version is not needed/suggested.

Also you should check in Amz23 at least systemd startup settings as those are somehow different than in RHEL. Cgroups default is v2 which needs some parameter changes etc. Also if your environment needs IMDS its version has changed to v2. Probably doesn’t affect to you unless yo are using some old AWS ta?

[venksel]

Hi @PickleRick @isoutamo 

Thanks for taking time to reply. Much appreciated. I'd prefer to take the upgrade path of 8.2->9.0.x -> 9.2.7; unfortunately, 9.0 does not support AL2023.

Please take a moment to review the migration log file as didn't see any alarming (you may download from the link below), i did disable THP and verified the ulimits to match the recommended settings.

https://limewire.com/d/PDWiS#NfyxSpwkrX

Also,  log ingestion is working properly . It's been a week since I upgraded our test instance of Splunk to 9.27 and there has been no issues. The health stats from Settings->Monitoring Console reports no issues. And we dont use workload management feature in Splunk from a cgroup compatibility standpoint we are covered...

With regards to SystemD, it was setup using the below commands in AL2023:

sudo /opt/splunk/bin/splunk enable boot-start -user splunk -systemd-managed 1

sudo systemctl daemon-reload

sudo systemctl start splunkd

sudo systemctl enable splunkd

[PickleRick]

As I said before - if it worked, good for you.

And I'm not gonna go through your logs. For several reasons.

Don't take it personally but this is a public forum where people give time whenever they want, the way they want. Answers is not a free support/consulting service. If you want something specific done - reach out to your local Splunk Partner or hire Professional Services. To put it bluntly - it costs money.

Secondly, you've already done the migration and apparently you're running the new system as prod. My looking into your logs won't change anything.

Thirdly, I might miss something, especially considering the first point.

You asked, you have been advised, you did otherwise. I won't pat you now on the back and say "it's ok". It might. But don't have to. And I don't have spare time just to ease your mind, to be frank.

I hope it is ok. But I won't check it.

[LAME-Creations]

You are going to have to contact Splunk Support for any older versions not on their website.  I apologize for that inconvenience.  

It is your environment and you need to do what you and your management team feel are the best things, but as a person employed in the Cyber Security arena, I feel that I should at least mention the following.  None of this applies to your wanting to run 9.0.x  It was the Splunk 7 and Splunk 8 that raised my antennae.  

Running 7.x (and to a lesser extent 8.x) UFs introduces significant risks, especially since Splunk 7.x reached End of Support (EOS) between October 2020 and October 2021, and 8.2.x is also at or past end of life. Here are the key implications:

• Operational Risks:
  • Limited Functionality: Splunk 7.x UFs lack support for newer features like data compression, advanced SSL configurations, or Splunk-to-Splunk (S2S) Protocol V4, which 9.x indexers use by default. This can cause performance issues or data ingestion failures if configurations mismatch. For example, 7.x UFs may not handle modern event-breaking or parsing rules in 9.0.x apps.
  • Management Challenges: If you use a Deployment Server (DS), it must be 9.0.x or newer to manage 7.x/8.x UFs. Older DS versions may fail to deploy apps to newer UFs, complicating configuration management.
  • Stability Issues: 7.x UFs may encounter bugs or crashes on modern OSes (e.g., newer Linux kernels), as they were designed for older environments. Splunk Support won’t provide fixes for EOS versions, leaving you to work around issues manually.
• Security Risks:
  • Vulnerabilities: 7.x UFs miss critical security patches available in 8.x and 9.x, exposing your environment to known vulnerabilities (e.g., CVE fixes). Without patches, UFs could be exploited, especially if they’re on internet-facing systems or handle sensitive data.
  • SSL/TLS Weaknesses: 7.x UFs use outdated SSL/TLS protocols, which may conflict with 9.0.x’s stricter security defaults (e.g., TLS 1.2/1.3). This can lead to connection failures or insecure data transmission. 8.x UFs are less problematic but still lack the latest TLS enhancements in 9.x.
  • Compliance Issues: Running EOS software like 7.x may violate compliance requirements (e.g., PCI DSS, HIPAA), as auditors often flag unsupported software as non-compliant.

Recommendations for UFs:

• Upgrade UFs to 9.0.x: Plan to upgrade your 7.x and 8.x UFs to 9.0.x (or at least 8.2.x) to align with your indexer. UFs are lightweight, and upgrades are straightforward 
  • Start with a few test UFs to validate compatibility with your 9.0.x indexer and DS (if used).
  • Use the Deployment Server to automate UF upgrades, ensuring serverclass.conf matches the new version.
• Prioritize 7.x First: 7.x UFs are the most critical to upgrade due to EOS status and severe security risks. 8.x UFs are less urgent but should be updated to avoid future EOS issues.
• Check Compatibility: Confirm UF OS compatibility with 9.0.x (e.g., AL2023 or supported Windows versions) using the Splunk System Requirements.
• Interim Step: If upgrading all UFs immediately isn’t feasible, ensure your 9.0.x indexer’s inputs.conf supports legacy S2S protocols (e.g., V3 for 7.x UFs) by setting connectionTimeout or readTimeout to accommodate older clients. However, this is a temporary workaround.

Why Upgrade UFs?:

• Aligning UFs with 9.0.x ensures optimal performance, security, and supportability. Splunk 9.0.x introduces features like ingest actions and enhanced TLS validation, which 7.x UFs can’t leverage.
• Upgrading avoids the risk of data loss or ingestion delays due to protocol mismatches or unpatched bugs.
• Splunk Support can assist with 9.0.x issues, but not with 7.x, reducing your troubleshooting burden.

[PickleRick]

Actually, compression has been around for quite a long time and 7.x forwarders should support it.

Also, your protocol levels are way off.

Not to mention the bogus requirement to use 9.0+ DS to manage 7/8 version UFs.

Please refrain from posting AI-generated content.

[PickleRick]

And why would you go for 9.0 which is out of support? I'd strongly advise against that. Unless you have a very good reason for doing so (and a very very good support contract, other than us, mere mortals) it's unwise to keep your environment at an unsupported version (which applies to the current 8.2 as well).

[richgalloway]

Contact Splunk Support for versions not available on the web site.