Deployment Architecture

Does a light forwarder constantly resolve output server hostnames?


We are engaged in a project that will involve pointing our light forwarders to new Splunk servers. We list hostnames instead of IP addresses in our outputs.conf files on the light forwarders. Note that in this case, the outputs.conf does auto-loadbalancing between 2 Splunk servers.

Part of the plan is to make a DNS change that alters the IP address of the hosts listed in outputs.conf. However, I realize that if somehow Splunk resolves those hostnames only once -- at startup time -- and then never attempts to resolve them again, this is going to be a problem. It means we'd have to restart each forwarder to pick up the changed IP addresses for the output servers.

If, as we hope, Splunk resolves the hostnames to IP addresses each time it uses them, this means that the light forwarders will just pick up the change shortly after it occurs in DNS.

Does anyone happen to know how the light forwarder would handle name resolution in this case?



You could try using port forwarding (use something like ipchains if your indexer is on a linux box).

The process I have in mind:

  1. Bring up the new splunk indexer.
  2. Shut down the old splunk indexer. (requests start to queue on the forwarders)
  3. Start a (logging) port forwarder on the old splunk indexer server which forwards the splunk port to the new indexer. (requests all start going to the new indexer)
  4. Do the IP address change.
  5. After the port forwarder stops forwarding any packets, shut down your old indexer server.
0 Karma

Splunk Employee
Splunk Employee

If outputs.conf is already configured for auto-load balancing, why not just use one of the supported load balancing methods? You can have Splunk alternate IP addresses for a FQDN by simply configuring multiple A records for the FQDN on the DNS server. You can use autoLBFrequency to determine how often Splunk will rotate servers.

Here's a verbose description of what your forwarder can do and how to configure it:

Your method seems difficult to support and maintain in the long run. Generally speaking, when an application opens a socket using a hostname, a resolver will attempt various methods to translate the name into an IP address. The resolver is not part of the application, but a part of the IP stack and/or OS. If resolution was provided by a DNS server, there will be an associated TTL for the DNS entry. The TTL specifies that any resolver needs to cache the DNS entry for a specified maximum period of time. This doesn't just affect the client asking for the resolution -- it is also cached by the DNS server answering the question. Unless you use a single DNS server that is also authoritative for the zone, there is no reliable method to predict when a client will actually receive an updated DNS entry, since any intermediary DNS servers will also need to expire their cache.

Splunk Employee
Splunk Employee

Automatic Load Balancing with Splunk uses multiple A records mapped to a hostname. CNAMEs (aliases) won't produce the desired effect. IMO, use this method to configure a single hostname to LB your indexers, expire the A records for whichever IP you will retire, then migrate, then add an A record for the new indexer to add its IP to the LB pool.

0 Karma


So if I understand your suggestion, you're saying that I should create some new CNAMEs for our end-of-lease refresh servers and make our existing autoLB pool a 4-way instead of a 2-way? We take the old servers down while we copy the data (no Splunk indexing during this period), then bring up the new indexers only and the forwarders will send to them? That at some point, revert the 4-way LB config on the forwarder back to a 2-way so that they don't timeout 50% of the time? Thanks.

0 Karma


Right. And my concern was that we have some other common vendor apps that will not-resolve a name after the app starts up. The only way to have that app detect an IP address change is to restart it. This is really about handling a server that's at end of lease. However, we'd like to reuse the CNAME that's in place for the host on the new host.

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes
and swag!