Getting Data In

Issue with light forwarder on cloned hosts

emiller42
Motivator

We recently added several hosts that would be forwarding data to our indexers. Since all the hosts were going to be cloned off of a template, we thought that to save time, we would simply install the forwarder on the template, give it the proper deploymentclient.conf. That way, when we cloned it, the clones would simply poll the deployment server and be up and running without any manual intervention.

This seemed to work, as the clones did pull the appropriate application bundles from the deployment server. But we aren't seeing forwarded data on the indexer. Checking splunkd.log on the forwarders shows errors where the indexer would refuse the connection, but we can't see any reason why. All configs are correct, and there are no connectivity blocks between the forwarder and the indexer.

Any ideas of what to check next?

Tags (2)
0 Karma
1 Solution

emiller42
Motivator

Answering this one myself in case it helps someone else.

I just discovered that the installer for the light forwarder creates an inputs.conf in etc/system/local that specifies the host name. Since we ran the installer on the template, this .conf file had the host name of the template. Then when we cloned the template, all of the clones had that conf file in place, setting their host name to be identical to the template.

Since the connection refusals from the indexer were due to it getting what appeared to be 30+ simultaneous connections from the same host.

Modifying the conf file to have the appropriate host name and restarting the forwarder corrected the issue.

View solution in original post

emiller42
Motivator

Answering this one myself in case it helps someone else.

I just discovered that the installer for the light forwarder creates an inputs.conf in etc/system/local that specifies the host name. Since we ran the installer on the template, this .conf file had the host name of the template. Then when we cloned the template, all of the clones had that conf file in place, setting their host name to be identical to the template.

Since the connection refusals from the indexer were due to it getting what appeared to be 30+ simultaneous connections from the same host.

Modifying the conf file to have the appropriate host name and restarting the forwarder corrected the issue.

emiller42
Motivator

Good info, thanks!

0 Karma

kristian_kolb
Ultra Champion

you might also ensure that GUID is unique for all the cloned forwarders, since that is what e.g. the DeploymentMonitor uses to separate the forwarders from each other. I learned this the hard way 🙂

http://splunk-base.splunk.com/answers/32368/duplicate-guids-for-cloned-forwarders-how-to-correct

Also there is also a section of the docs relating to cloning; http://docs.splunk.com/Documentation/Splunk/latest/Admin/PutSplunkontosystemimages and the pages that follow it.

/k

Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...