Deployment Architecture

Replacing Immutable Splunk Forwarders show as duplicates in Deployment Server

aDevOpsPerson
Engager

We have a 5 node Splunk forwarder cluster to handle throughput of multiple servers in our datacenter.  Currently our upgrade method is keeping the the Deployment server as mutable where we just run config. changes via Chef, and update it.  But, the 5 node forwarders are being treated as fully replaceable with Terraform and Chef.

Everything is working, but I notice the Deployment server holds onto forwarders after Terraform destroys the old one, and the new one pings home on a new IP(currently on DHCP), but with the same hostname as the destroyed forwarder. 

Would replacing the forwarders with the same static IP and Hostname resolve that, or would there still be duplicate entries?

Deployment server:
Oracle Linux 8.10
Splunk-enterprise 8.2.9

Forwarders:
Oracle Linux 8.10
Splunkforwarder 8.2.9

Labels (3)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

The Deployment Server tracks forwarders by GUID rather than by name and/or address.  Each time Splunk is installed it generates a new GUID, which is why you see the same host name multiple times.

To retain forwarder info across rebuilds, save and restore the $SPLUNK_HOME/etc/instance.cfg file on each forwarder.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Message Parsing in SOCK

Introduction This blog post is part of an ongoing series on SOCK enablement. In this blog post, I will write ...

Exploring the OpenTelemetry Collector’s Kubernetes annotation-based discovery

We’ve already explored a few topics around observability in a Kubernetes environment -- Common Failures in a ...

Use ‘em or lose ‘em | Splunk training units do expire

Whether it’s hummus, a ham sandwich, or a human, almost everything in this world has an expiration date. And, ...