How to set up a search head connection to an index cluster via a Network Address Translation (NAT) device?

[acidkewpie]

Hi,

I'm designing a deployment where there will be a search head on the other side of a NAT boundary to an index cluster. In order to fully connect the search head to the cluster, the SH will connect to the cluster master and poll back details about all the indexers, exchanging keys etc. As I understand it at least, you can't trivially just point the SH at a cluster.

As such, the addresses that the SH will then query are the same addresses that are used within the cluster. So if raw IP's are used, then that will be what the SH tries to hit. If there is a NAT boundary between the devices, those actual addresses are not reachable. So am I right in believing that if the cluster is assembled using local hostnames that are locally resolvable within the cluster (e.g. in /etc/hosts), then on the SH alternative host entries can be provided (again in /etc/hosts) which can relate to the addresses on the NAT boundary, allowing end to end connectivity to happen?

Alternatively... am I thinking this is more complex than it is on the clustering side and you can just tell the SH to hit an alternative IP?

Additionally, if this boundary device is HTTP aware, would splunk care if the unique hostnames resolved to the same IP address? Here we can then use the HTTP Host header to connect the SH to the right indexer in a scalable way. (This is all configured with SSL disabled BTW, so I can now see the raw API calls trivially)

Thanks

[tabab]

Messing with IP's won't help because often you'd have one public IP on the NAT with multiple public ports that are translated to multiple private IP's on port 8089. Something like this:
- pub_ip:pub_port_1 -> priv_ip_1:8089
- pub_ip:pub_port_2 -> priv_ip_2:8089

What you need is to rewrite destination IP and port on your search heads to point to PUBLIC IP and port. Here is how:

#!/bin/bash

PATH=/bin:/usr/bin:/user/local/bin:/sbin:/usr/sbin:/usr/local/sbin

REMOTE_PUB_IP=<your_public_ip_here>
REMOTE_PUB_PORT=(8089 8090 8091)

REMOTE_PRV_IP=(your_private_ips_here, separated by space)
REMOTE_PRV_PORT=(8089 8089 8089)


run_cmd () {
    if [[ -z "$DEBUG" ]]; then
        $*
    else
        echo $*
    fi
}


# Enable IP forwarding
sysctl net.ipv4.ip_forward=1

# Flush all NAT rules
iptables -t nat -F

(( max_index = ${#REMOTE_PUB_PORT[*]} - 1 ))
for i in $( seq 0 $max_index ); do
    run_cmd "iptables -t nat -A PREROUTING -p tcp --dest ${REMOTE_PRV_IP[i]} --dport ${REMOTE_PRV_PORT[i]} -j DNAT --to ${REMOTE_PUB_IP}:${REMOTE_PUB_PORT[i]}"
    run_cmd "iptables -t nat -A OUTPUT     -p tcp --dest ${REMOTE_PRV_IP[i]} --dport ${REMOTE_PRV_PORT[i]} -j DNAT --to ${REMOTE_PUB_IP}:${REMOTE_PUB_PORT[i]}"
done

Anybody knows how to do that using firewall-cmd?