Getting Data In

Issue with light forwarder on cloned hosts

emiller42
Motivator

We recently added several hosts that would be forwarding data to our indexers. Since all the hosts were going to be cloned off of a template, we thought that to save time, we would simply install the forwarder on the template, give it the proper deploymentclient.conf. That way, when we cloned it, the clones would simply poll the deployment server and be up and running without any manual intervention.

This seemed to work, as the clones did pull the appropriate application bundles from the deployment server. But we aren't seeing forwarded data on the indexer. Checking splunkd.log on the forwarders shows errors where the indexer would refuse the connection, but we can't see any reason why. All configs are correct, and there are no connectivity blocks between the forwarder and the indexer.

Any ideas of what to check next?

Tags (2)
0 Karma
1 Solution

emiller42
Motivator

Answering this one myself in case it helps someone else.

I just discovered that the installer for the light forwarder creates an inputs.conf in etc/system/local that specifies the host name. Since we ran the installer on the template, this .conf file had the host name of the template. Then when we cloned the template, all of the clones had that conf file in place, setting their host name to be identical to the template.

Since the connection refusals from the indexer were due to it getting what appeared to be 30+ simultaneous connections from the same host.

Modifying the conf file to have the appropriate host name and restarting the forwarder corrected the issue.

View solution in original post

emiller42
Motivator

Answering this one myself in case it helps someone else.

I just discovered that the installer for the light forwarder creates an inputs.conf in etc/system/local that specifies the host name. Since we ran the installer on the template, this .conf file had the host name of the template. Then when we cloned the template, all of the clones had that conf file in place, setting their host name to be identical to the template.

Since the connection refusals from the indexer were due to it getting what appeared to be 30+ simultaneous connections from the same host.

Modifying the conf file to have the appropriate host name and restarting the forwarder corrected the issue.

emiller42
Motivator

Good info, thanks!

0 Karma

kristian_kolb
Ultra Champion

you might also ensure that GUID is unique for all the cloned forwarders, since that is what e.g. the DeploymentMonitor uses to separate the forwarders from each other. I learned this the hard way 🙂

http://splunk-base.splunk.com/answers/32368/duplicate-guids-for-cloned-forwarders-how-to-correct

Also there is also a section of the docs relating to cloning; http://docs.splunk.com/Documentation/Splunk/latest/Admin/PutSplunkontosystemimages and the pages that follow it.

/k

Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Using the Splunk Threat Research Team’s Latest Security Content

REGISTER HERE Tech Talk | Security Edition Did you know the Splunk Threat Research Team regularly releases ...

SplunkTrust | 2024 SplunkTrust Application Period is Open!

It's that time again, folks! That's right, the application/nomination period for the 2024 SplunkTrust is ...