Indexer Web UI is not available after migrating Cluster master hardware

[_pravin]

Hi,

Our Linux machine has reached the End of Support, so we are moving the Cluster Master from one machine to another.

I set up the cluster master in the new hardware and it was working well, but when I changed the master node URL in the indexer it was not working.

The indexer doesn't turn on by itself and even when I turn it on manually, the indexer stays running for some time but during that time the web UI of the indexer does not work. In some time the indexer stops automatically. The same happened for another indexer as well.

When I revert to the old cluster master, all the issues are sorted automatically. Splunk indexer always keeps running, web UI is available. No issues are noticed.

Any idea why the indexer keeps shutting down?

I am Splunk version 9.0.4

Regards,

Pravin

[isoutamo · Accepted solution]

Can you describe how you have done this migration to the new master? There are several ways to do this and some works better than another. Here is one which I have used successfully. https://community.splunk.com/t5/Splunk-Enterprise/Migration-of-Splunk-to-different-server-same-platf...

[_pravin]

Hi @isoutamo ,

I followed a similar strategy and I'll list it below:

1. Set up new CM, and turn on indexer clustering.
2. Turn off the instance and copy  'etc/master-apps' and 'etc/manager-apps'
3. Setup server.conf in the new instance with relevant changes ( pass4symkey in string format)
4. Start the new CM Change the URL of the indexer cluster one by one.
5. Finally, change the URL in the search head after the indexers are migrated.

This had worked in the test environment but when it was time for the production setup, the indexers failed to connect and would keep stopping after changing to new CM.

Regards,

Pravin

[isoutamo]

Did you copy all those files and directories which are mentioned on item 2.4 in referenced post? I’m not sure how it works and are there some additional stuff to do as you have changed the name/url for the new master? I prefer to use FQDN as all instance names (CNAME or A records) to avoid additional issues which can arise when there are too many changes at same time.
Is your old master still available to test with it?
Are there any reasonable error messages on CM’s or indexers’ logs to get more information what is the issue?

[_pravin]

I didn't copy all the files and directories mentioned by you and also didn't put the old CM in maintenance mode but did change the URL and FDQN for all the instances. Probably the problem arose from the fact that the production instance has a lot of moving data and not going into maintenance mode caused the problems. Even, the test site had moving data but did not have a lot of data like production.

What was surprising is that there were no logs that showcase any exact reason of error.

However, I used the techniques mentioned by you and was able to migrate CM to new hardware.

Thanks to you @isoutamo 

[dural_yyz]

The indexer and the new CM will have logs to help indicate what is happening, something to point you in the right direction.  Please look there and post anything of interest if you still need help after reviewing.

As for the webURL on the indexer, IMO from a security stance should always be disabled.  Your environment so hopefully there is a good reason for that.

Knowing that the webURL availability turns on and off does tell me that your old CM has a custom app that enables webURL, the new CM likely does not so when the new CM pushes a bundle the indexer removes the oldCM custom app and disables the webURL.