Getting Data In

Issue with light forwarder on cloned hosts

emiller42
Motivator

We recently added several hosts that would be forwarding data to our indexers. Since all the hosts were going to be cloned off of a template, we thought that to save time, we would simply install the forwarder on the template, give it the proper deploymentclient.conf. That way, when we cloned it, the clones would simply poll the deployment server and be up and running without any manual intervention.

This seemed to work, as the clones did pull the appropriate application bundles from the deployment server. But we aren't seeing forwarded data on the indexer. Checking splunkd.log on the forwarders shows errors where the indexer would refuse the connection, but we can't see any reason why. All configs are correct, and there are no connectivity blocks between the forwarder and the indexer.

Any ideas of what to check next?

Tags (2)
0 Karma
1 Solution

emiller42
Motivator

Answering this one myself in case it helps someone else.

I just discovered that the installer for the light forwarder creates an inputs.conf in etc/system/local that specifies the host name. Since we ran the installer on the template, this .conf file had the host name of the template. Then when we cloned the template, all of the clones had that conf file in place, setting their host name to be identical to the template.

Since the connection refusals from the indexer were due to it getting what appeared to be 30+ simultaneous connections from the same host.

Modifying the conf file to have the appropriate host name and restarting the forwarder corrected the issue.

View solution in original post

emiller42
Motivator

Answering this one myself in case it helps someone else.

I just discovered that the installer for the light forwarder creates an inputs.conf in etc/system/local that specifies the host name. Since we ran the installer on the template, this .conf file had the host name of the template. Then when we cloned the template, all of the clones had that conf file in place, setting their host name to be identical to the template.

Since the connection refusals from the indexer were due to it getting what appeared to be 30+ simultaneous connections from the same host.

Modifying the conf file to have the appropriate host name and restarting the forwarder corrected the issue.

emiller42
Motivator

Good info, thanks!

0 Karma

kristian_kolb
Ultra Champion

you might also ensure that GUID is unique for all the cloned forwarders, since that is what e.g. the DeploymentMonitor uses to separate the forwarders from each other. I learned this the hard way 🙂

http://splunk-base.splunk.com/answers/32368/duplicate-guids-for-cloned-forwarders-how-to-correct

Also there is also a section of the docs relating to cloning; http://docs.splunk.com/Documentation/Splunk/latest/Admin/PutSplunkontosystemimages and the pages that follow it.

/k

Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...