The other day I came across universal forwarder based deployment client which was not receiving deployment server apps as expected.
When looking at the HttpPubSubConnection (PhoneHome) records in the splunkd log on the deployment client I noticed that the IP address posted to the deployment server was an IP address of one of the host's network interfaces which did not have a route to the deployment server. When I restarted the splunk agent, the next phone home happened to post the correct IPv4 address having a valid route to the deployment server, and of course the apps downloaded.
How does the deployment client decide which ipv4 address (of possibly multiple) to present when phoning home?
2018-01-24: Initial submission
To best of my knowledge it's based on Routing table present on OS.
For example: You have 2 interface eth0 and eth1, eth0 does not have route to Deployment Server on Switch/Firewall & eth1 has route present to pass traffic to Deployment Server and default route passing traffic from eth0 on OS and Deployment Server does not belong to eth0 or eth1 subnet.
In this case Forwarder will not able to fetch apps from Deployment Server because it will initiate connection from Forwarder to Deployment Server on default route which is eth0 and that subnet does not have route present to Deployment Server.
The client IP is based on what the tcp layer of the server sees, it could be the NAT or proxy address instead of the actual client ip.