Migrating Single-site Indexer Cluster Splunk Enterprise from Windows to Linux

[atizon]

Greetings,

I need help to understand which steps I have to take in order to migrate my Splunk environment from Windows to Linux, trying to minimize downtime.

I'm currently working on a single-site indexer cluster environment, consisting of the following machines, all of them running on Windows:
1 Master node
2 Indexers (peer nodes)
1 Search Head

Also, the indexes are stored on NAS, each indexer is conected to the NAS through an iSCSI connection.

I need to replace the OS of the four machines mentioned above form Windows to Linux, preseving all the indexed data and my configurations (apps, dashboards, alerts, etc.) and trying to minimize the downtime to avoid data loss. If posible, I would like to reuse the IPs on the new environment to avoid configuration changes in other hosts.

If I understood the documentation correctly, I shoud do the folowing steps:
1) Remove all the nodes from the cluster.
2) Stop Splunk Enterprise on the diferent nodes.
3) Copy the $SPLUNK_HOME directory from the Windows host to the Linux host.
4) Install Splunk on the Linux hosts.
5) Change the paths on the configuration files.
6) Change the iSCSI connection to the new indexers.
7) Start Splunk Enterprise on the new host.
8) Rebuild the indexer cluster.

Is this the best way to proceed, or is there a better solution? Would it be possible to prepare the new Linux environment while the Windows one is still active, and when it's ready, connect the new indexers to the NAS to access the indexed data? If so, which steps showd be carried out?

Thank you.

[woodcock]

1) The first step is to add the new indexers newidx1 and newidx2 to your environment mimicking exactly the configuration of the existing indexers. This is not solely the cluster-bundle, you should look for existing apps that aren’t part of the base Splunk package.

2) Next, you will place your old indexers in detention (link to Da Xu’s talk below). Then, you will direct your legacy (oldidx1, oldidx2) indexers to offline --enforce-counts to shut them down gracefully, which will trigger the CM to fix these buckets over to the new hardware.

Read the PDF carefully, as well as look in docs. This is a live production system, so exercise prudence and caution. It is in your best interests to not rush the steps, and ensure each node is fully dealt with before calling the lab complete. Finally, docs provides detailed instruction for post-offline steps as well. Consider performing these steps as well.
http://conf.splunk.com/files/2016/slides/indexer-clustering-internals-scaling-and-performance.pdf

[atizon]

Thanks for your response. I had read similar answers looking for a solution, but I thought that I couldn't just add the new indexers in my environment, since, according to the Splunk documentation, in "System requirements and other deployment considerations for indexer clusters", all machines that are part of the indexer cluster must have the same operating system, therefore I couldn't add new indexers running on Linux while my master node and other indexers are on Windows. Can I enable the Linux indexers on my current environment without any issue? If so, since I also what to replace the master and search head, should I do that before or after replacing the indexers?

[woodcock]

That directive is referring to a supported operational state. You are entering into a supported migration state. It is absolutely fine to have a multi-OS Indexer cluster temporarily for the purposes of migration. The Search Head node migration is much more trivial since it can be run simultaneously. I absolutely would NOT upgrade the CM before the Indexers, unless you are planning to upgrade the version of Splunk (CM always has to be higher version).

[atizon]

Thanks again. Although I think the solution you proposed is the best, I'm not sure if it can be implemented in my environment. As I said in my first question, indexers store the indexes on a NAS via iSCSI connections, which gives me two doubts.

On one hand, I can't create the new indexers with the same configuration as the old ones, because on the NAS I only have two LUNs, each one with an iSCSI connection, so I can't have the new and old indexers connected to the NAS at the same time for replication, and I need that after the migration, the new indexers keep saving the indexes on the NAS. I'm investigating whether it would be possible to resize the current LUNs to create two new ones, and be able to replicate.

On the other hand, due to the way disks are mounted in Windows and Linux, the indexes path would be different in each operating system, but because those paths are defined on the indexes.conf files on the master node, i can't specify the rigth path for each indexer.

Is there any way to set different index paths on each indexers, or to temporarily change the path during migration in order to solve these problems?

Would it be possible to directly connect the new indexers to the LUNs, so that they already have all the indexes without doing buket fixup between the old and new indexers, even if that means stopping data collection for a day or so?

[woodcock]

IMHO, you would be WAY better of using Direct-Attached-Storage for the new Indexers and ditching NAS.