We have several Universal Forwarders installed on different Linux machines. Due to the virtualization technology, each of the Linux servers has several ip addresses. By default the Universal Forwarder uses the first one (eth0 on this example). I assume this happens because the UF just asks the OS for opening the connection without specifying the interface to be used.
eth0 Link encap:Ethernet HWaddr XX:XX:XX:XX inet addr:XX:XX:XX:XX Bcast:XX:XX:XX:XX Mask:255.255.254.0 inet6 addr: XX:XX:XX:XX/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:10680683134 errors:0 dropped:11120 overruns:0 frame:0 TX packets:8692419851 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:3381271414547 (3224631.7 Mb) TX bytes:3873093410263 (3693669.7 Mb) eth0:0 Link encap:Ethernet HWaddr XX:XX:XX:XX inet addr:YY:YY:YY:YY Bcast:XX:XX:XX:XX Mask:255.255.254.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Due to firewall restrictions we need to use a (secondary/virtual) different ip address for the outgoing connections (eth0:0 YY.YY.YY.YY on the example). We didnt find any clue on the documentation about how to achieve this behavior. Any idea?
Many thanks in advance!
Your issue is not really a Splunk problem.
The requirement to route traffic for a specific app over a specific nic, is poorly defined.
Given the your response to the suggestion above, you seem to be trying something which is in conflict with the way all Operating Systems handle networking.
If you want to segregate Splunk traffic to a specific Nic, you should do so by properly segmenting your network, not by changing settings on your hosts.
If you wish to do this properly, you should run your Splunk infrastructure in a separate vlan, and put an interface for that vlan on each of your hosts. The OS will then correctly route requests over the correct (sub)interface keeping your UF traffic over the dedicated Nic.
Thanks for your reply. Some comments:
a) The network topology and the virtualization technology is something that doesnt depend on me. So there is nothing I can change there.
b) Many other software products (even SAP Netweaver or a simple Linux telnet or netcat) have the option via parameter to bind the outgoing connections to a specific interface or ip address. So perhaps its not such a crazy idea.
A) of course, I didn’t mean to imply you were at fault. It’s clear from your description that you have been given a set of requirements you are attempting to work to them.
B.) all the applications in your comment are server applications. You are correct, it is quite common for server apps to listen on a specific Ip or interface, however the Splunk Universal Forwarder is a client.
It figures out the correct network interface and route based on the address of the server it is connecting to. (Hence my suggestion about network segregation)
You can’t configure your web browser or email client to use a specific Ip/interface- it’s the address of the website/mail server which dictates that.
B) No no, Im talking about the telnet client (so using the -b option) and the netcat client. Also when speaking about SAP the parameter "icm/local_addr" for the ICM affects to the connections as client and does not apply to any server socket.
For sure you are right and this is a requierement that is mandatory for all server applications, but also quite common for client applications. Regarding browsers and email clients I have no idea if this is possible somehow on the configuration, but maybe the requirement is not so common and there is no such possibility.
At the risk of straying too far off the point, the examples you have listed are a bit special.
netcat is a utility more than an application, its also both a server & client. You can do as you say, but netcat is a tool which is often used to 'work-around' network limitations, or remotely access something otherwise not possible. - More on this in a second..
telnet has the ability to bind to a specific network configuration (interface) this is a legacy feature which stems from a time when an IP address was considered something you could authenticate with (oh how we laugh these days 🙂
Both of the above are often used today as 'tools' because they allow you to do somewhat odd things, just as you cite.
I have no knowledge of SAP, so can't comment on its use case for this feature.
To use your netcat example though, you could use it to solve your issue.
Create a netcat listener on localhost on port 9089 & 9097 on each of your UFs
Configure netcat to establish outbound connections using your desired interface, and forward traffic to your HF/IDX on port 8089/9997
In this way, the UF outbound traffic will pass through your local network stack, into netcat, and then out of your desired interface.
I cant recommend that this is a good idea, but it does illustrate what netcat can be used for.
Ok, thanks for your explanations. I still think this is a nice feature for clients working on a scenario full of firewalls. So Im afraid we will not agree on that 😉
BTW thanks again for the discussion.
You're welcome, good luck with a solution!
If you do find a way to achieve it, please post back here so others can see how you did it.
Yes, this can be done. Though the change would be on the OS side, not Splunk.
You just need to add a new route and assign it to the device you for that traffic.
ip route add YY:YY:YY:YY/24 dev eth1
Verify with "ip route list" or "ip route show"
To make the change persist across reboots you'll need to add an entry to /etc/sysconfig/network-scripts/route-eth1, e.g.
Many thanks for your quick and valuable reply. Unfortunately this solution doesnt fulfill our requirements. The virtualization technology I mentioned on my original post enables us to move the whole system to a different phisycal box so any fix at OS level is not valid. As soon as the application is moved to a different phisical host, the ip route will not be there and we are again at the beginning :S
@codebuilder means make the change on the Splunk Host VM (not the hypervisor)
That way when you migrate the VM, the route change will persist
It wont affect other traffic
YY:YY:YY:YY/24 refers to your indexer - so it will only affect traffic sent to indexer(s)
But you would have to do it for every host.