Getting Data In

Is it possible to force an Universal Forwarder to use an specific ip address for the connection to the indexer/hf?

ayoldi
Explorer

We have several Universal Forwarders installed on different Linux machines. Due to the virtualization technology, each of the Linux servers has several ip addresses. By default the Universal Forwarder uses the first one (eth0 on this example). I assume this happens because the UF just asks the OS for opening the connection without specifying the interface to be used.

Linux ifconfig:

eth0      Link encap:Ethernet  HWaddr XX:XX:XX:XX  
          inet addr:XX:XX:XX:XX  Bcast:XX:XX:XX:XX  Mask:255.255.254.0
          inet6 addr: XX:XX:XX:XX/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:10680683134 errors:0 dropped:11120 overruns:0 frame:0
          TX packets:8692419851 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:3381271414547 (3224631.7 Mb)  TX bytes:3873093410263 (3693669.7 Mb)

eth0:0    Link encap:Ethernet  HWaddr XX:XX:XX:XX  
          inet addr:YY:YY:YY:YY  Bcast:XX:XX:XX:XX  Mask:255.255.254.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

Due to firewall restrictions we need to use a (secondary/virtual) different ip address for the outgoing connections (eth0:0 YY.YY.YY.YY on the example). We didnt find any clue on the documentation about how to achieve this behavior. Any idea?

Many thanks in advance!

Regards,

0 Karma

nickhills
Ultra Champion

Your issue is not really a Splunk problem.

The requirement to route traffic for a specific app over a specific nic, is poorly defined.
Given the your response to the suggestion above, you seem to be trying something which is in conflict with the way all Operating Systems handle networking.

If you want to segregate Splunk traffic to a specific Nic, you should do so by properly segmenting your network, not by changing settings on your hosts.

If you wish to do this properly, you should run your Splunk infrastructure in a separate vlan, and put an interface for that vlan on each of your hosts. The OS will then correctly route requests over the correct (sub)interface keeping your UF traffic over the dedicated Nic.

If my comment helps, please give it a thumbs up!

ayoldi
Explorer

Thanks for your reply. Some comments:

a) The network topology and the virtualization technology is something that doesnt depend on me. So there is nothing I can change there.
b) Many other software products (even SAP Netweaver or a simple Linux telnet or netcat) have the option via parameter to bind the outgoing connections to a specific interface or ip address. So perhaps its not such a crazy idea.

Regards,

0 Karma

nickhills
Ultra Champion

Hi,
A) of course, I didn’t mean to imply you were at fault. It’s clear from your description that you have been given a set of requirements you are attempting to work to them.

B.) all the applications in your comment are server applications. You are correct, it is quite common for server apps to listen on a specific Ip or interface, however the Splunk Universal Forwarder is a client.
It figures out the correct network interface and route based on the address of the server it is connecting to. (Hence my suggestion about network segregation)
You can’t configure your web browser or email client to use a specific Ip/interface- it’s the address of the website/mail server which dictates that.

If my comment helps, please give it a thumbs up!

ayoldi
Explorer

B) No no, Im talking about the telnet client (so using the -b option) and the netcat client. Also when speaking about SAP the parameter "icm/local_addr" for the ICM affects to the connections as client and does not apply to any server socket.

For sure you are right and this is a requierement that is mandatory for all server applications, but also quite common for client applications. Regarding browsers and email clients I have no idea if this is possible somehow on the configuration, but maybe the requirement is not so common and there is no such possibility.

0 Karma

nickhills
Ultra Champion

At the risk of straying too far off the point, the examples you have listed are a bit special.

netcat is a utility more than an application, its also both a server & client. You can do as you say, but netcat is a tool which is often used to 'work-around' network limitations, or remotely access something otherwise not possible. - More on this in a second..

telnet has the ability to bind to a specific network configuration (interface) this is a legacy feature which stems from a time when an IP address was considered something you could authenticate with (oh how we laugh these days 🙂

Both of the above are often used today as 'tools' because they allow you to do somewhat odd things, just as you cite.

I have no knowledge of SAP, so can't comment on its use case for this feature.

To use your netcat example though, you could use it to solve your issue.

Create a netcat listener on localhost on port 9089 & 9097 on each of your UFs
Configure netcat to establish outbound connections using your desired interface, and forward traffic to your HF/IDX on port 8089/9997

In this way, the UF outbound traffic will pass through your local network stack, into netcat, and then out of your desired interface.

I cant recommend that this is a good idea, but it does illustrate what netcat can be used for.

If my comment helps, please give it a thumbs up!
0 Karma

ayoldi
Explorer

Ok, thanks for your explanations. I still think this is a nice feature for clients working on a scenario full of firewalls. So Im afraid we will not agree on that 😉

BTW thanks again for the discussion.

0 Karma

nickhills
Ultra Champion

You're welcome, good luck with a solution!
If you do find a way to achieve it, please post back here so others can see how you did it.

~Cheers, Nick

If my comment helps, please give it a thumbs up!

codebuilder
SplunkTrust
SplunkTrust

Yes, this can be done. Though the change would be on the OS side, not Splunk.

You just need to add a new route and assign it to the device you for that traffic.

For example:

ip route add YY:YY:YY:YY/24 dev eth1

Verify with "ip route list" or "ip route show"
To make the change persist across reboots you'll need to add an entry to /etc/sysconfig/network-scripts/route-eth1, e.g.

----
An upvote would be appreciated and Accept Solution if it helps!

ayoldi
Explorer

Many thanks for your quick and valuable reply. Unfortunately this solution doesnt fulfill our requirements. The virtualization technology I mentioned on my original post enables us to move the whole system to a different phisycal box so any fix at OS level is not valid. As soon as the application is moved to a different phisical host, the ip route will not be there and we are again at the beginning :S

BTW thanks!

0 Karma

nickhills
Ultra Champion

@codebuilder means make the change on the Splunk Host VM (not the hypervisor)
That way when you migrate the VM, the route change will persist

If my comment helps, please give it a thumbs up!
0 Karma

codebuilder
SplunkTrust
SplunkTrust

Yes, exactly. vMotion will keep the IP intact.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

ayoldi
Explorer

We are using SAP LVM and we cannot perform the route operation on all hosts (hundreds) because it will also impact other systems that are not using the Splunk UF.

0 Karma

nickhills
Ultra Champion

It wont affect other traffic
YY:YY:YY:YY/24 refers to your indexer - so it will only affect traffic sent to indexer(s)
But you would have to do it for every host.

If my comment helps, please give it a thumbs up!
0 Karma

codebuilder
SplunkTrust
SplunkTrust

Sure, no problem. What virtualization technology are you referring to? VMWare?

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...