Following the steps in this document: http://docs.splunk.com/Documentation/Splunk/6.2.5/Installation/MigrateaSplunkinstance
This is Linux to Linux - Prior to doing this on the new hardware, we are attempting to do this in the lab environment in which we built and configured an identical machine to use as a new indexer. The above document does not appear to cover everything needed to do this.
We shut down the lab/test system completely and rsynced the data. We have some indexes pointing to /splunkdatahot /splundatacold... frozen... etc. And a couple still sitting in main in the /opt/splunk/var/lib/splunk file system. So /opt/splunk, /opt/splunk/var/lib/splunk, /splunkdatahot /spunkdatacold, /splundatafrozen, and splundatamodels were copied from the old VM to the new VM to test the process. While this lab environment is V to V, the process should be identical for V to P if I understand things correctly.
When things first came up, the new indexer needed the --accept-license flag as expected. Had to change some of the configuration files (add the new name to the indexer whitelist in the server class and chase down where the machine name for http://i[machine]:8000/en-US/app/launcher/home#en-US/app/launcher/home was stored. We are able to search the copied data.
We have our custom forwarder outputs file where the indexer name is stored (in prod this will use the alias and we will repoint the alias and not have to do this) and we changed the name of the name of the tcpout:primary_indexers stanza to the new machine name. Deployment server pushed that out to the new indexer (the only machines running at this point are the indexer and the deployment server).
We are getting an error message on the indexer console:
Forwarding to indexer group primary_indexers blocked for 100 seconds.
Since this machine is the only machine in the that group so we don't know why it's saying that.
forwarder outputs looks like this:
[tcpout]
defaultGroup = primary_indexers
[tcpout:primary_indexers]
server = [servername]006.[fqdn]:9997
We see the forwarder outputs being pushed out to the universal forwarders with that name in them but we have no new data coming into the index. The forwarder logs are also showing the forwarder still attempting to talk to the old indexer name even though forwarder_outputs contains the new indexer name.
In short, we cannot do this in prod unless we have a much clearer 'how to' guide. If anyone can help us fill in the blanks we would greatly appreciate it.
