Replacing Immutable Splunk Forwarders show as duplicates in Deployment Server

[aDevOpsPerson]

We have a 5 node Splunk forwarder cluster to handle throughput of multiple servers in our datacenter.  Currently our upgrade method is keeping the the Deployment server as mutable where we just run config. changes via Chef, and update it.  But, the 5 node forwarders are being treated as fully replaceable with Terraform and Chef.

Everything is working, but I notice the Deployment server holds onto forwarders after Terraform destroys the old one, and the new one pings home on a new IP(currently on DHCP), but with the same hostname as the destroyed forwarder. 

Would replacing the forwarders with the same static IP and Hostname resolve that, or would there still be duplicate entries?

Deployment server:
Oracle Linux 8.10
Splunk-enterprise 8.2.9

Forwarders:
Oracle Linux 8.10
Splunkforwarder 8.2.9

[richgalloway]

The Deployment Server tracks forwarders by GUID rather than by name and/or address.  Each time Splunk is installed it generates a new GUID, which is why you see the same host name multiple times.

To retain forwarder info across rebuilds, save and restore the $SPLUNK_HOME/etc/instance.cfg file on each forwarder.