Getting Data In

Issue with light forwarder on cloned hosts

emiller42
Motivator

We recently added several hosts that would be forwarding data to our indexers. Since all the hosts were going to be cloned off of a template, we thought that to save time, we would simply install the forwarder on the template, give it the proper deploymentclient.conf. That way, when we cloned it, the clones would simply poll the deployment server and be up and running without any manual intervention.

This seemed to work, as the clones did pull the appropriate application bundles from the deployment server. But we aren't seeing forwarded data on the indexer. Checking splunkd.log on the forwarders shows errors where the indexer would refuse the connection, but we can't see any reason why. All configs are correct, and there are no connectivity blocks between the forwarder and the indexer.

Any ideas of what to check next?

Tags (2)
0 Karma
1 Solution

emiller42
Motivator

Answering this one myself in case it helps someone else.

I just discovered that the installer for the light forwarder creates an inputs.conf in etc/system/local that specifies the host name. Since we ran the installer on the template, this .conf file had the host name of the template. Then when we cloned the template, all of the clones had that conf file in place, setting their host name to be identical to the template.

Since the connection refusals from the indexer were due to it getting what appeared to be 30+ simultaneous connections from the same host.

Modifying the conf file to have the appropriate host name and restarting the forwarder corrected the issue.

View solution in original post

emiller42
Motivator

Answering this one myself in case it helps someone else.

I just discovered that the installer for the light forwarder creates an inputs.conf in etc/system/local that specifies the host name. Since we ran the installer on the template, this .conf file had the host name of the template. Then when we cloned the template, all of the clones had that conf file in place, setting their host name to be identical to the template.

Since the connection refusals from the indexer were due to it getting what appeared to be 30+ simultaneous connections from the same host.

Modifying the conf file to have the appropriate host name and restarting the forwarder corrected the issue.

emiller42
Motivator

Good info, thanks!

0 Karma

kristian_kolb
Ultra Champion

you might also ensure that GUID is unique for all the cloned forwarders, since that is what e.g. the DeploymentMonitor uses to separate the forwarders from each other. I learned this the hard way 🙂

http://splunk-base.splunk.com/answers/32368/duplicate-guids-for-cloned-forwarders-how-to-correct

Also there is also a section of the docs relating to cloning; http://docs.splunk.com/Documentation/Splunk/latest/Admin/PutSplunkontosystemimages and the pages that follow it.

/k

Get Updates on the Splunk Community!

Best Strategies to Optimize Observability Costs

 Join us on Tuesday, May 6, 2025, at 11 AM PDT / 2 PM EDT for an insightful session on optimizing ...

Fueling your curiosity with new Splunk ILT and eLearning courses

At Splunk Education, we’re driven by curiosity—both ours and yours! That’s why we’re committed to delivering ...

Splunk AI Assistant for SPL 1.1.0 | Now Personalized to Your Environment for Greater ...

Splunk AI Assistant for SPL has transformed how users interact with Splunk, making it easier than ever to ...