Can universal forwarders have 2 different deployment server IPs in case the first fails?

[StephenD1]

I have UFs in the DMZ and internal networks with a load balancer managing traffic between both zones. There is a single deployment server located in the internal network.

Up to this point we had the network team open port 8089 for all DMZ servers to connect directly to the deployment server in the internal network. We're changing this and instead will route traffic through the load balancer to the deployment server. 

However, if possible, I want to keep both routes so that if one IP fails, the UF will try the second IP. Both IPs will point to the same deployment server so this is not technically a "DS cluster" it's just 2 different routes to the same server. My drawing below shows what I would want to do.

Would it be possible to input multiple target-brokers or targetUris in deploymentclient.conf?
If this is not allowed in the config, would it be possible to use an FQDN instead with a DNS record pointing to multiple IPs? 

Any advice would help, even if it's "this isn't a good idea". 

P.S. I know a load balancer is not recommended, removing it is a no-go.

[livehybrid]

Hi @StephenD1 

Personally I would use DNS here for an active/passive setup where you can change the DNS to point to the online server. 

As others have mentioned, since 9.2 there has been better support DS high availability by using a load balancer so that could be an option for you?

Alternatively you could look at using something like Keepalived if you are comfortable setting it up and managing it, this would provide a single IP which could float between your DS depending on availability. 

Ultimately "it depends" and the best option really depends on your full architecture and comfort with other non-Splunk specific architectures/approaches.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[PrewinThomas · Accepted solution]

@StephenD1 

Straight forward answer is no. Splunk does not support multiple targetUri entries or multiple [target-broker] stanzas for failover or load balancing. The forwarder will only connect to one deployment server URI, and if that route fails, it won’t automatically try a backup.

#https://help.splunk.com/en/data-management/splunk-enterprise-admin-manual/9.1/configuration-file-ref...

I recommend, A simpler, well-monitored DS with good backup and version control is often more effective than trying to engineer failover.
Regards,
Prewin
If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!

[PickleRick]

Adding to what @isoutamo already said.

Historically, there was a possibility to create a "kinda-HA" DS solution based on  multiple DS servers behind a load-balancer. It required some careful planning though and a specfic set of configuration on DS side as well as proper administrative procedures. Since that was kinda tricky, it was not a widely shared knowledge. But this could be achieved.

Since 9.2 there have been changes to DS architecture allowing you to implement a DS cluster natively. See https://docs.splunk.com/Documentation/Splunk/9.4.2/Updating/Implementascalabledeploymentserversoluti... for details.

Oh, and remember that:

1) As others already pointed out - DS is not essential for your forwarders to run. They will continue running even if the DS fails completely.

2) For performance reasons you might (and probalby want) to increase the phonehome interval so the threshold for "I need multiple DS instances to handle the load" raises significantly.

[gcusello]

Hi @StephenD1 ,

in addition to the perfect answer of @kiran_panchavat , I whould add only one information: forwarders continue to work also without a running Deplyment Server, so you don't need to cluster it, you could eventually use the VM-Ware solution described by Kiran, but it isn't mandatory.

Ciao.

Giuseppe

[kiran_panchavat]

@StephenD1  Deployment clients only talk to one Deployment Server URI. There’s no built-in failover, and there’s no DS clustering/HA. If the DS is down or unreachable, UFs keep running with the last downloaded configs and retry on the next phone-home.

If you really need HA on your DS, the best option is to use VMWare. Clone your primary DS and leave it down, then bring it back up if you need it.

Another (probably better) option is to store / version control your DS configs in GitHub (or similar repo) so that you can quickly spin up another instance in case you have issues with the primary.

https://lantern.splunk.com/Splunk_Success_Framework/Platform_Management/Preparing_for_failures_in_th... 

https://docs.splunk.com/Documentation/Splunk/9.4.2/Admin/Deploymentclientconf#.5Btarget-broker:deplo... 

[StephenD1]

I just wanted to clarify that I'm not afraid necessarily of the server itself going down but that the route will be unavailable due to network issues outside of my control. So I still want to have only one deployment server but I was hoping that if, for example, the load balancer went down, the UF would still be able to talk to the DS via the secondary route. It seems, per your response, this is not possible.
Perhaps this is outside the scope of a Splunk forum but my follow-up to this is that the targetUri in the deploymentclient.conf.spec does take an FQDN pointing to the DS instead of an IP. Would a DNS related solution work? I could register both the load balancer IP and the direct DS IP as A records. Would this cause any issues with the UFs being able to reach the DS?

[PickleRick]

If you want to manage multiple routes, that's wrong layer you're looking at. For this you have all the network-level protocols - OSPF, BGP, whatever.

If you want to have a HA LB, do HA LB. If you want DNS-failover, do DNS-failover, UFs don't care. They just use system calls to pull content from the DS.

So you're mixing different things here.

BTW, if you do have a proxy and DS reachable directly, why use the proxy at all?

[isoutamo]

This is not the whole truth. Even you don’t need DS for log collection and you can run your environment some time without DS, you could also create “HA” version of DS. 
With current DS versions 9.2+ it’s architecture has changed to support it. 
You should read more https://help.splunk.com/en/splunk-enterprise/administer/update-your-deployment/9.4/configure-the-dep...