Splunk Search

How to set up a search head connection to an index cluster via a Network Address Translation (NAT) device?

Path Finder


I'm designing a deployment where there will be a search head on the other side of a NAT boundary to an index cluster. In order to fully connect the search head to the cluster, the SH will connect to the cluster master and poll back details about all the indexers, exchanging keys etc. As I understand it at least, you can't trivially just point the SH at a cluster.

As such, the addresses that the SH will then query are the same addresses that are used within the cluster. So if raw IP's are used, then that will be what the SH tries to hit. If there is a NAT boundary between the devices, those actual addresses are not reachable. So am I right in believing that if the cluster is assembled using local hostnames that are locally resolvable within the cluster (e.g. in /etc/hosts), then on the SH alternative host entries can be provided (again in /etc/hosts) which can relate to the addresses on the NAT boundary, allowing end to end connectivity to happen?

Alternatively... am I thinking this is more complex than it is on the clustering side and you can just tell the SH to hit an alternative IP?

Additionally, if this boundary device is HTTP aware, would splunk care if the unique hostnames resolved to the same IP address? Here we can then use the HTTP Host header to connect the SH to the right indexer in a scalable way. (This is all configured with SSL disabled BTW, so I can now see the raw API calls trivially)


0 Karma

New Member

Messing with IP's won't help because often you'd have one public IP on the NAT with multiple public ports that are translated to multiple private IP's on port 8089. Something like this:
- pub_ip:pub_port_1 -> priv_ip_1:8089
- pub_ip:pub_port_2 -> priv_ip_2:8089

What you need is to rewrite destination IP and port on your search heads to point to PUBLIC IP and port. Here is how:



REMOTE_PUB_PORT=(8089 8090 8091)

REMOTE_PRV_IP=(your_private_ips_here, separated by space)
REMOTE_PRV_PORT=(8089 8089 8089)

run_cmd () {
    if [[ -z "$DEBUG" ]]; then
        echo $*

# Enable IP forwarding
sysctl net.ipv4.ip_forward=1

# Flush all NAT rules
iptables -t nat -F

(( max_index = ${#REMOTE_PUB_PORT[*]} - 1 ))
for i in $( seq 0 $max_index ); do
    run_cmd "iptables -t nat -A PREROUTING -p tcp --dest ${REMOTE_PRV_IP[i]} --dport ${REMOTE_PRV_PORT[i]} -j DNAT --to ${REMOTE_PUB_IP}:${REMOTE_PUB_PORT[i]}"
    run_cmd "iptables -t nat -A OUTPUT     -p tcp --dest ${REMOTE_PRV_IP[i]} --dport ${REMOTE_PRV_PORT[i]} -j DNAT --to ${REMOTE_PUB_IP}:${REMOTE_PUB_PORT[i]}"

Anybody knows how to do that using firewall-cmd?

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In the last month, the Splunk Threat Research Team (STRT) has had 2 releases of new security content via the ...

Announcing the 1st Round Champion’s Tribute Winners of the Great Resilience Quest

We are happy to announce the 20 lucky questers who are selected to be the first round of Champion's Tribute ...

We’ve Got Education Validation!

Are you feeling it? All the career-boosting benefits of up-skilling with Splunk? It’s not just a feeling, it's ...