What are the steps for Splunk enterprise migration physical to virtual?

[Cloudangelo]

Hello 

I am planning to migrate Splunk Enterprise from a physical server(RHEL7) to a VM(RHEL8). 

On the new VM, I already installed the latest version of Splunk Enterprise ( 9.0.5).  The old instance Splunk enterprise version is 8.0.2.

What are the steps to perform this migration?  Will I run into conflicts if I will jump versions since it's not in place upgrade? 

I have checked the following  I could not find anything related physically to virtual migration documentation https://docs.splunk.com/Documentation/Splunk/latest/Installation/MigrateaSplunkinstance

Should I  move $SPLUNK_HOME and then proceed with the new Splunk install? o

[Cloudangelo]

Thank you, everyone. I decided to migrate to the same version 8.0.3 in source and destination following @richgalloway advice and followed @isoutamo  migration steps:

- setup new host
- rsync (1st - Nth time) data + configs
- yum install splunk
- test with new version w/o alarms, emails etc.
- stop old
- final rsync with delete option
- start new

After I started the Splunkd service in the new instance I got some  ERRORS few about SSL certificate which I think got resolved (as I don't see it in /ops/splunk/var/splunk/splunk.log)  by generating a new SSL certificate and signing them using the organization CA 

and ..

ERROR TailReader - File will not be read, is too small to match seekptr checksum (file=/opt/splunk/var/log/splunk/migration.log.2023-06-26.17-32-08). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.
06-27-2023 14:18:08.891 -0400 ERROR IntrospectionGenerator:resource_usage - RU - Mount '/opt/splunk' (ext4) is not interesting, iostats will not be collected.

And 

-0400 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value

[isoutamo]

As you haven’t rsync/copy old logs etc (you don’t need those) splunk will warn as it expect to find it based on fishbucket. You could ignore that warning. Probably there are some other too?

If you didn’t rsync in 1st time whole SPLUNK_HOME/etc you have only trial license etc which you should change to the correct one. Syncing the etc directory gives to you also same GUID and splunk.secret which you have on old environment. This ensures that your old passwords etc are still working.

[Cloudangelo]

I did rsync both SPLUNK_HOME and SPLUNK_DB ( entirely ) I had to change the server name in some of the *.conf  files to the new one

[Cloudangelo]

I am still not able to see the GUI 

[isoutamo]

Have you ensure that all files under SPLUNK_HOME are owned by splunk OS user?

Are there anything else on splunkd.log or migration log if you have already done it?

Have you done all needed OS level changes based on your OS?

[Cloudangelo]

for the ownership I ran 

chown -R splunk:splunk /app/splunk-app/($SPLUNK_HOME) and same for /app/splunk-data ($SPLUNK_DB)

there is a symlink /app/splunk-app to /opt/splunk (to enable RPM installs and updates).

for the OS just installed the RPMs with yum localinstall <RPM>

in regards to  other errors :

ERROR DatabaseDirectoryManager - Getting size on disk: Path for bid=<index>~5614~E1F35BEE-29E7-4CCB-9E16-F000A9A1741F cannot be located. caller=serialize_SizeOnDisk

migration log: /opt/splunk/var/log/splunk/migration.log.2023-06-26.17-32-08:

Migrating to:

VERSION=8.0.3

BUILD=a6754d8441bf

PRODUCT=splunk

PLATFORM=Linux-x86_64

Copying '/opt/splunk/etc/myinstall/splunkd.xml' to '/opt/splunk/etc/myinstall/splunkd.xml-migrate.bak'.

Checking saved search compatibility...

Handling deprecated files...

Checking script configuration...

Copying '/opt/splunk/etc/myinstall/splunkd.xml.cfg-default' to '/opt/splunk/etc/myinstall/splunkd.xml'.

Deleting '/opt/splunk/etc/system/local/field_actions.conf'.

The following apps might contain lookup table files that are not exported to other apps:

splunk_monitoring_console

Such lookup table files could only be used within their source app.  To export them globally and allow other apps to access them, add the following stanza to each /opt/splunk/etc/apps/<app_name>/metadata/local.meta file:

[lookups]

export = system

For more information, see http://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/SetPermissions#Make_objects_globally_....

Checking for possible UI view conflicts...

 App "splunk_monitoring_console" has an overriding copy of the "alerts.xml" view, thus the new version may not be in effect. location=/opt/splunk/etc/apps/splunk_monitoring_console/default/data/ui/views

 App "splunk_monitoring_console" has an overriding copy of the "dashboards.xml" view, thus the new version may not be in effect. location=/opt/splunk/etc/apps/splunk_monitoring_console/default/data/ui/views

 App "splunk_monitoring_console" has an overriding copy of the "reports.xml" view, thus the new version may not be in effect. location=/opt/splunk/etc/apps/splunk_monitoring_console/default/data/ui/views

Removing legacy manager XML files...

Removing legacy nav XML files...

DMC is not set up, no need to migrate nav bar.

Removing System Activity dashboards...

Removing splunkclouduf XML file...

Removing splunkclouduf view XML files...

Distributed Search is not configured on this instance

Removing legacy search.xml file from splunk_instrumentation...

Deleting '/opt/splunk/share/splunk/search_mrsparkle/modules'.

Moving '/opt/splunk/share/splunk/search_mrsparkle/modules.new' to '/opt/splunk/share/splunk/search_mrsparkle/modules'.

Checking for the modules related files and folders that should not be present after upgrade.

Checking for the Advanced XML dashboard templates that should not be present after upgrade.

Checking for the 'Getting Started' app that should not be present after upgrade.

It seems that the Splunk default certificates are being used. If certificate validation is turned on using the default certificates (not-recommended), this may result in loss of communication in mixed-version Splunk environments after upgrade.

"/opt/splunk/etc/auth/ca.pem": already a renewed Splunk certificate: skipping renewal

"/opt/splunk/etc/auth/cacert.pem": already a renewed Splunk certificate: skipping renewal

Clustering migration already complete, no further changes required.

Generating checksums for datamodel and report acceleration bucket summaries for all indexes.

If you have defined many indexes and summaries, summary checksum generation may take a long time.

Processed 1 out of 12 configured indexes.

Processed 2 out of 12 configured indexes.

Processed 3 out of 12 configured indexes.

Processed 4 out of 12 configured indexes.

Processed 5 out of 12 configured indexes.

Processed 6 out of 12 configured indexes.

Processed 7 out of 12 configured indexes.

Processed 8 out of 12 configured indexes.

Processed 9 out of 12 configured indexes.

Processed 10 out of 12 configured indexes.

Processed 11 out of 12 configured indexes.

Processed 12 out of 12 configured indexes.

Finished generating checksums for datamodel and report acceleration bucket summaries for all indexes.

[App Key Value Store migration] NOTE: Lock file is not empty. If migration fails, stop the service (if it is running), delete the lock file, and run the migration again.

[App Key Value Store migration] Checking if migration is needed. Upgrade type 1. This can take up to 600 seconds.

[App Key Value Store migration] Migration is not required.

[App Key Value Store migration] Checking if migration is needed. Upgrade type 2. This can take up to 600 seconds.

[App Key Value Store migration] Migration is not required.

[DFS] Performing migration.

[DFS] Finished migration.

[isoutamo]

To be on the same page, we should check that we have common understanding.

1. You have updated old env to 8.0.3
2. You have installed RH 8 into the new server
3. You have rsync /opt/splunk/ and db dir to the new server
4. You have changed a needed conf files etc
5. You have installed splunk 8.0.3 into the new server
6. You have started splunk on new sever
7. now we are here?

[Cloudangelo]

1. You have updated old env to 8.0.3 - The current(old) Splunk version is 8.0.3. In the destination(new instance), the version is the same version and I am planning to do in-place upgrade to 8.1  to~> 9.0.5 ( once migrated successfully )
2. You have installed RH 8 into the new server- ( our company provides the instances the new one is RedHat8 the old was 7) 
3. You have rsync /opt/splunk/ and db dir to the new server - correct
4. You have changed a needed conf files etc - correct to contain the new server name (FQDN)
5. You have installed splunk 8.0.3 into the new server- correct
6. You have started splunk on new sever -correct
7. now we are here? Yes,sir 

[isoutamo]

One question. On RH 8 there is usually firewalld in use. Have you opened needed ports with it? Also some SElinux changes could be needed?

And what you are actually meaning when you are saying “I cannot see GUI”?

[Cloudangelo]

The GUI means the Splunk dashboard where we log in and run a query

Are you referring: https://www.splunk.com/en_us/blog/tips-and-tricks/selinux-and-splunk.html

[isoutamo]

As you have SElinux disabled then it's not a problem here.

What you get when you are running as root

[root@splunk-demo-rh8] ~>
(0) # systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2023-05-26 15:32:05 EEST; 1 months 2 days ago
     Docs: man:firewalld(1)
 Main PID: 1055 (firewalld)
    Tasks: 2 (limit: 100406)
   Memory: 36.8M
   CGroup: /system.slice/firewalld.service
           └─1055 /usr/libexec/platform-python -s /usr/sbin/firewalld --nofork --nopid

May 26 15:32:03 splunk-demo-rh8.local systemd[1]: Starting firewalld - dynamic firewall daemon...
May 26 15:32:05 splunk-demo-rh8.local systemd[1]: Started firewalld - dynamic firewall daemon.
May 26 15:32:06 splunk-demo-rh8.local firewalld[1055]: WARNING: AllowZoneDrifting is enabled. This is considered an insecure configuration option. It will be removed in a future release. Please consider di>
[root@splunk-demo-rh8] ~>
(0) # firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: ens3
  sources:
  services: cockpit dhcpv6-client ssh
  ports: 8000/tcp 8089/tcp 9997/tcp 8088/tcp
  protocols:
  forward: no
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:
[root@splunk-demo-rh8] ~>
(0) #

If firewalld is enabled as here, you should have opened those ports (or what ever you are using) to get access to GUI + REST api and sending events from UF + enable HEC. 

[Cloudangelo]

I don't have Firewalld installed 

[isoutamo]

Try systemctl instead of systmctl

[Cloudangelo]

[isoutamo]

Ok, can you connect to server or did it refused a connection?

Can you you see your connection attempts on *access.logs?

[Cloudangelo]

from splunkd.log :

06-28-2023 12:12:09.162 -0400 WARN HttpListener - Socket error from <ip> while idling: error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http reques

[Cloudangelo]

I found the issue.. 

1) I got confused as the previous instance DNS pointer is not updated to the new FQDN, so when I tried to hit the DNS name it did not hit the new instance 

2) The new instance was configured with web.conf ->  enableSplunkWebSSL = 1  to enforce SSL but I think there is an issue with the common name

[gcusello]

Hi @Cloudangelo,

I'd like only to add a little contribution to the perfect answer from @richgalloway:

passing from physical to virtual, you have also to check the performaces of the storage that must be at least 800 IOPS (better 1200), check that resources are dedicated to Splunk (not shared), and add around 30% of additional resources to your virtual machines that the resources you have on physical.

Especially storage performances are mandatory to have a good global system performance: storage is the Splunk bottleneck!

Ciao.

Giuseppe

[isoutamo]

Hi

I totally agree with @gcusello and @richgalloway the way you should do it. Definitely install old version onto a new server then update. If/when you have a single node installation or idx + sh separately you could do it quite easily. Here is a old post how to do it https://community.splunk.com/t5/Installation/How-to-migrate-indexes-to-new-indexer-instance/m-p/5280...

 If you have distributed environment then you should just add new nodes to clusters and do the migration with splunk cluster features like it has told on this post https://community.splunk.com/t5/Splunk-Enterprise/Migration-of-Splunk-to-different-server-same-platf.... And again after migration to new nodes do a normal upgrade for distributed environment.

r. Ismo 

[richgalloway]

A physical-to-virtual migration is no different than a physical-to-physical migration other than the additional considerations for the virtual machine itself (see https://docs.splunk.com/Documentation/Splunk/latest/Capacity/Referencehardware#Virtualized_Infrastru...).

I advise against migrating to a new version.  Better to migrate to the same version then upgrade or upgrade then migrate.  Upgrading separately ensures any necessary file changes are made.

Be sure to read the Release Notes before upgrading, especially for version 9.0 because there are several potentially breaking changes in 9.  Also, you must be running 8.1 to upgrade to 9.