Archive

Splunk forwarder is using wrong NIC

jamesklassen
Path Finder

Setting up a CAS server for the Exchange app. There are two NICs on the machine (2008 R2), the deployment server is seeing the private IP for the CAS array. Yet the binding order for the NICs are to use the public interface before the private one. Not sure why Splunk appears to be using the private IP to communicate with the deploy server. How do I tell Splunk to use the public interface?

I've restarted the Splunk forwarding service, and reloaded the deploy-server.

0 Karma

gpullis
Communicator

So, your "private" and "public" IP addresses are on the same subnet? In that case, you'll need to add a static route that points to your Splunk server and specifies which interface to use:

Do a

netsh interface ip show interface

to get your interface indexes (Idx column). Then, do a:

netsh interface ip add route 1.2.3.4/32 interface=1 nexthop=142.16.7.1

Where 1.2.3.4 is the IP address of your Splunk deployment server, 1 is the interface index you got from the show interface command, and 142.16.7.1 is the gateway for the 142.16.7.0/24 subnet. Modify appropriately with your real world values.

You'll probably want to set up a static route to point to your indexer too.

You can to a netsh interface ip add route /? to get help for the command.

0 Karma

gpullis
Communicator

For the one that works, my guess is that you just got lucky as it had a 50/50 chance of picking the one you wanted. It's probably too late for this, but it may have behooved you to have your private and public IPs on separate subnets.

0 Karma

jamesklassen
Path Finder

It does indeed look like a routing problem. The servers, including the deployment server, are on 142.16.7.0. For a CAS server that's properly identifying itself, the routing looks like:

   142.16.7.0    255.255.255.0         On-link      142.16.7.141    266
   142.16.7.0    255.255.255.0         On-link       142.16.7.21    266

For the problemantic CAS server, it has this:

   142.16.7.0    255.255.255.0         On-link       142.16.7.20    266
   142.16.7.0    255.255.255.0         On-link      142.16.7.142    266

I assume the issue is that the private IP comes first on the problematic CAS. Any idea how to fix that? (aware it's not a splunk issue). I'm unable to reboot this server without 3 days of Change Management notification to users, so a fix with no impact to production users would be good.

gkanapathy
Splunk Employee
Splunk Employee

I'm not sure exactly how you'd have to fix it, but in Windows you can specifiy cost/priorities for different interfaces, which will affect the order in which routing is chosen. The CLI route add command can let you change it, but you should probably actually set it via the TCP/IP Networking Control Panel.

0 Karma

dwaddle
SplunkTrust
SplunkTrust

Is this a binding question, or more of an addressing/routing one? Unless configured with something like SPLUNK_BINDIP, Splunk relies on the OS to decide the appropriate interface to use based on the destination address and the route table.

I would confirm the address/DNS name for the deployment server in deploymentclient.conf, and check the routing table to see which interface should be selected for that address/name. If the same name DNS maps to multiple IP addresses, this might contribute to the issue.

Of course, if you are using SPLUNK_BINDIP then all the above is moot.