Migrated system to new server, now indexes show as empty

[gillisme]

Hi,

Not sure if this is even a problem, but thought I would be proactive and ask what other folks have experienced. 

*Note* that I am note an experienced splunk admin - I can do a few things, add users, add forwarders, and have updated to newer version, but I really don't know how to use it. Admin newby.

We are running splunk enterprise v 8.2.6, on a single RHEL6 server. We need to get off of RHEL6, so plan was to migrate the splunk install to a new RHEL8 server, and then upgrade to newest splunk version. My understanding of splunk is that it is pretty self contained - to update the version, you just overwrite the /opt/splunk dir with the new splunk tar file. Our data is held in a separate filesystem, /data/splunk dir.

So, the process was:

1- install splunk v 8.2.6 on the new rhel8 server, and verify it starts and works

2- shutdown old rhel6 splunk

3- copy the old rhel6 /opt/splunk dir on top of the new rhel8 /opt/splunk dir

4- copy the old rhel6 /data/splunk dir on to the new rhel8 server, in the /data/splunk dir

5- shutdown the rhel6 splunk server

6- ensure all the networking, DNS, etc is resolving to the new rhel8 server

7- start up splunk on the new rhel8 server

The process was followed by me this morning, and appears to have worked. I am seeing forwarders (there are 160) check in on the new server, I can run searches on host=X and see that X has been in contact. But there is one thing I am seeing, that I don't know if it is a problem or not. If I look at "Indexes and Volumes: Instance" for the previous 24 hours, there is data there up until the old rhel6 server was turned off. Since moving to the new rhel8 server, the indexes all appear to be 0GB in size.

I don't know enough to know whether this is an issue. It seems like it is, to me, but I am not really sure - could everything just be rebuilding on the new server, or has it become unavailable somehow?

If anyone has an answer I would be glad to know. Otherwise I find out Monday morning, I guess, when the users log on to the new rhel8 server.

Thanks, Michael.

[gillisme]

Thanks for the replies.

Please note that there was some planning in doing this, that the new system had been up for a few months, and that the splunk+data was rsynced over several times and tested. We obviously failed on the testing aspect, I have a single contact for users of this application, and relied on their reports of everything looking OK. Our system is only used weekdays, so after the Saturday change and realization that things were amiss, resulted in turning off the new server and reverting to the original server, so no unplanned production outage, system is running as it was before.

I have to say (from inventsekars' reply) that in step 3, it is not immediately clear to me that I am to reinstall splunk on top of my original /opt/splunk directory rsynced over from the existing system. I would have thought that all the existing config files would be overwritten, and it would then seem to be like a new install. That is why I installed 8.2.6 splunk on the new server first, and then rsynced the existing data on top of it, to ensure any configuration files were intact. 

I am opening a ticket with Splunk today, to go over the process and investigate why the data was not reachable by the application. I appreciate the suggestions, but I do not know how to "check _internal index for errors" or do "tstats". Maybe I should request my employer send me for Splunk admin training, if they are expecting my to administer it.

Cheers, Michael.

[PickleRick]

The way it works with installations/upgrades is that the installation package contains binaries and default settings which you should not touch. Ever. You only edit your local configs. That way if you deploy a new software version (whether it's by means of rpm upgrade or simply untaring the archive over your old installation) it should replace the default settings but leave your installation-specific local settins intact.

That's why it's always stressed - don't touch the files residing in etc/system/default (the same goes for the apps - apps come with their own default directory which you should not touch since it'll get overwritten with app upgrade and you create config settings in the local directory).

[isoutamo]

Only exception to don’t edit files inside default folder is, when you have created your own app. Then you must edit default files before deploying it to splunk. But then with gui you edit again only local files. 

[PickleRick]

Well, no. That's not how you do it.

1. I know it's not gonna help here but you forgot to do one very important thing as this is something you're not experienced with - plan and test. You should have deployed a test instance (possibly using a trial version of Splunk), then do your planned migration and verify if everything is OK.

2. You say you have RHEL servers. Are you even using RPM packages? You should.

3. If the instructions say "unpack new version over your existing files", it's _not_ the same as unpack and then overwrite with your existing files.

At this moment it's hard to tell what config you ended up with and what's really happening underneath.

You can check _internal index for errors.

You can see whether your events are ingested into your indexes (for example by doing tstats over recent short period of time).

In this case you might simply have a misnamed server (many reports in MC search for host with your server's name - if you changed it, it could have caused some confusion for Splunk).

[isoutamo]

Hi

as @PickleRick already said, you must have a plan and test it before do it with production server!

Here is some old posts which told how you could do it. But if/when you already have done it these is just for next time 😞 Some of those also discuss what has done wrongly and how it could be fixed. But as said, it's really hard to guess what is the issue without seeing your environment. I also prefer to use rpm instead of tar on RH systems.

• https://community.splunk.com/t5/Installation/How-to-migrate-indexes-to-new-indexer-instance/m-p/5280...
• https://community.splunk.com/t5/Installation/What-are-the-steps-for-Splunk-enterprise-migration-phys...

r. Ismo

[inventsekar]

Hi @gillisme 

The Splunk Doc(selected your version 8.2.6) suggest to copy /opt/splunk(from old to new system) and then install the Splunk on the new system. (this is important, as when Splunk is installing, it checks the config files and it need to alter the installation depending on the config files)

https://docs.splunk.com/Documentation/Splunk/8.2.6/Installation/MigrateaSplunkinstance#How_to_migrat...

When you migrate on *nix systems, you can extract the tar file you downloaded directly over the copied files on the new system, or use your package manager to upgrade using the downloaded package. On Windows systems, the installer updates the Splunk files automatically.

1. Stop Splunk Enterprise services on the host from which you want to migrate.
2. Copy the entire contents of the $SPLUNK_HOME directory from the old host to the new host. Copying this directory also copies the mongo subdirectory.
3. Install Splunk Enterprise on the new host.
4. Verify that the index configuration (indexes.conf) file's volume, sizing, and path settings are still valid on the new host.
5. Start Splunk Enterprise on the new instance.
6. Log into Splunk Enterprise with your existing credentials.
7. After you log in, confirm that your data is intact by searching it.
   
   The 4th step -  4- copy the old rhel6 /data/splunk dir on to the new rhel8 server, in the /data/splunk dir is incorrect. As the data buckets in hot buckets should be treated carefully. Pls check this below steps from the doc:
   How to move index buckets from one host to another

   If you want to retire a Splunk Enterprise instance and immediately move the data to another instance, you can move individual buckets of an index between hosts, as long as:

   When you copy individual bucket files, you must make sure that no bucket IDs conflict on the new system. Otherwise, Splunk Enterprise does not start. You might need to rename individual bucket directories after you move them from the source system to the target system.

   1. Roll any hot buckets on the source host from hot to warm.
   2. Review indexes.conf on the old host to get a list of the indexes on that host.
   3. On the target host, create indexes that are identical to the ones on the source system.
   4. Copy the index buckets from the source host to the target host.
   5. Restart Splunk Enterprise.

   
   PS - if any reply helped you, pls upvote/add karma points. if any reply solves your query, pls accept it as solution, thanks.