We have HF 1 and HF2 that are located in DC1 and DC2 respectively. How can we configure outputs.conf in below method.
Please note that we are not looking for a autolb feature and just looking for a failover kind of configuration in outputs.conf
This is not how Splunk works, sorry.
Why are you using Heavy Forwarders? Any need to do so?
I don't really like to give you a workaround here, but.. If you have separate DNS in those data centres, you could manage a failover with DNS. So, in case server 1 goes down, assign the other IP address to its hostname.
I don't think Splunk explicitly can do this.
However, if you can introduce a load balancer in between and forward to the load-balanced URL, this is possible. Unfortunately, from your question, it looks like this is not one of the available options.
Splunk does not work that way but what you can do is do something like this:
server = DC1HF1:9997, DC2HF1:9999
server = DC2HF1:9997, DC1HF1:9999
Then always run inputs.conf with port
9997 on but
Then have cron job on both HFs that is checking on the health of his partner in the other DC and when it is down, enable the local
9999 port in
inputs.conf and restart local splunk. Do the opposite when the other side comes back up.