We are engaged in a project that will involve pointing our light forwarders to new Splunk servers. We list hostnames instead of IP addresses in our outputs.conf files on the light forwarders. Note that in this case, the outputs.conf does auto-loadbalancing between 2 Splunk servers.
Part of the plan is to make a DNS change that alters the IP address of the hosts listed in outputs.conf. However, I realize that if somehow Splunk resolves those hostnames only once -- at startup time -- and then never attempts to resolve them again, this is going to be a problem. It means we'd have to restart each forwarder to pick up the changed IP addresses for the output servers.
If, as we hope, Splunk resolves the hostnames to IP addresses each time it uses them, this means that the light forwarders will just pick up the change shortly after it occurs in DNS.
Does anyone happen to know how the light forwarder would handle name resolution in this case?
You could try using port forwarding (use something like ipchains if your indexer is on a linux box).
The process I have in mind:
If outputs.conf is already configured for auto-load balancing, why not just use one of the supported load balancing methods? You can have Splunk alternate IP addresses for a FQDN by simply configuring multiple A records for the FQDN on the DNS server. You can use
autoLBFrequency to determine how often Splunk will rotate servers.
Here's a verbose description of what your forwarder can do and how to configure it: http://www.splunk.com/base/Documentation/4.1.7/Admin/Setuploadbalancing
Your method seems difficult to support and maintain in the long run. Generally speaking, when an application opens a socket using a hostname, a resolver will attempt various methods to translate the name into an IP address. The resolver is not part of the application, but a part of the IP stack and/or OS. If resolution was provided by a DNS server, there will be an associated TTL for the DNS entry. The TTL specifies that any resolver needs to cache the DNS entry for a specified maximum period of time. This doesn't just affect the client asking for the resolution -- it is also cached by the DNS server answering the question. Unless you use a single DNS server that is also authoritative for the zone, there is no reliable method to predict when a client will actually receive an updated DNS entry, since any intermediary DNS servers will also need to expire their cache.
Automatic Load Balancing with Splunk uses multiple A records mapped to a hostname. CNAMEs (aliases) won't produce the desired effect. IMO, use this method to configure a single hostname to LB your indexers, expire the A records for whichever IP you will retire, then migrate, then add an A record for the new indexer to add its IP to the LB pool.
So if I understand your suggestion, you're saying that I should create some new CNAMEs for our end-of-lease refresh servers and make our existing autoLB pool a 4-way instead of a 2-way? We take the old servers down while we copy the data (no Splunk indexing during this period), then bring up the new indexers only and the forwarders will send to them? That at some point, revert the 4-way LB config on the forwarder back to a 2-way so that they don't timeout 50% of the time? Thanks.
Right. And my concern was that we have some other common vendor apps that will not-resolve a name after the app starts up. The only way to have that app detect an IP address change is to restart it. This is really about handling a server that's at end of lease. However, we'd like to reuse the CNAME that's in place for the host on the new host.