topic Re: Load Balancing UF to 3rd third party receivers in Getting Data In

Load Balancing UF to 3rd third party receivers

jknulst — Wed, 21 Oct 2020 17:55:17 GMT

Hi,

I have some troubles setting up the following topology. There is 1 UF which needs to forward unCooked raw data to a 3rd party receiver that is distributed and consists of 2 nodes.

[indexAndForward] index = false [tcpout:splunk-searchhead-group] disabled = false server = so1:9997 [tcpout-server://so1:9997] [tcpout-server://3rd_party_node_1:3535] [tcpout-server://3rd_party_node_2:3535] [tcpout] defaultGroup = splunk-searchhead-group [tcpout:default-autolb-group] disabled = false server = 3rd_party_node_1:3535,3rd_party_node_2:3535 sendCookedData = false forceTimebasedAutoLB = true autoLBVolume = 2 autoLBFrequency = 5 maxQueueSize = auto indexAndForward = false blockOnCloning = true compressed = false dropClonedEventsOnQueueFull = 5 dropEventsOnQueueFull = -1 heartbeatFrequency = 30 maxFailuresPerInterval = 2 secsInFailureInterval = 1 maxConnectionsPerIndexer = 2 connectionTimeout = 20 readTimeout = 300 writeTimeout = 300 tcpSendBufSz =

What happens in reality is that both 3rd_party_node_1 & 2 receive exactly the same data, it looks like data cloning in stead of load balancing.

Is there anything off in this config or is load balancing not possible with 3rd party receivers?

Thanks

Re: Load Balancing UF to 3rd third party receivers

nwuest — Tue, 27 Oct 2020 06:11:58 GMT

Hi @jknulst,

Your configuration looks good to me but needs one small tweak.

In your [tcpout] stanza you have the following:

[tcpout] defaultGroup = splunk-searchhead-group

I believe you need to add the other tcpout:group you have defined in your outputs.conf file so that the Universal Forwarder begins the "load-balancing" between the two 3rd-party indexers/nodes

[tcpout] defaultGroup = splunk-searchhead-group,default-autolb-group

Once you have set this, give your Splunk Universal Forwarder a reboot for the new configurations to take affect.
Your Splunk Universal Forwarder should now switch between each 3rd-party indexer/node every 30 seconds.

Note: If your box you have installed on is a linux variant, you might be able to run this command and see the switch happen in real-time:
# watch -d -n 0 "/opt/splunkforwarder/bin/splunk list forward-server"
You will be prompted for your Splunk U.F.'s username and password. Once entered, it should rerun that command and you will hopefully see the switchover in real-time.

If you don't have the "watch" command you can always tail the /opt/splunkforwarder/var/log/splunk/splunkd.log for the following command to see your Universal Forwarder switch nodes/indexers.
# tail -f /opt/splunkforwarder/var/log/splunk/splunkd.log | grep "TcpOutputProc"

If you are currently running on a windows box and have access to powershell you can use the following command to see the output of splunkd.log
(Open powershell in admin mode)
# Get-Content -Path "C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log" -Wait
It will chunk through the file until it gets to the end but

Please let us know if this solves your current challenge!

V/R,
nwuest

Re: Load Balancing UF to 3rd third party receivers

jknulst — Fri, 30 Oct 2020 20:50:50 GMT

@nwuest Thank you for your reponse and suggestion

Re: Load Balancing UF to 3rd third party receivers

jknulst — Fri, 30 Oct 2020 20:51:28 GMT

I followed your suggestion.

The output is as follows:

[root@uf1 splunkforwarder]# bin/splunk list forward-server Active forwards: 3rd_party_node_1:3535 3rd_party_node_2:3535 so1:9997 Configured but inactive forwards: None [root@uf1 splunkforwarder]#

And the tail:

[root@uf1 splunkforwarder]# tail -f /opt/splunkforwarder/var/log/splunk/splunkd.log | grep "TcpOutputProc" 10-30-2020 20:27:34.719 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0. 10-30-2020 20:27:39.613 +0000 INFO TcpOutputProc - After randomization, current is first in the list. Swapping with last item 10-30-2020 20:27:39.713 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1. 10-30-2020 20:27:43.720 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0. 10-30-2020 20:27:48.724 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1. 10-30-2020 20:27:52.722 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0. 10-30-2020 20:27:57.723 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1. 10-30-2020 20:27:59.335 +0000 INFO TcpOutputProc - Found currently active indexer. Connected to idx=172.19.0.4:9997, reuse=1. 10-30-2020 20:28:01.691 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0. 10-30-2020 20:28:06.719 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1. 10-30-2020 20:28:10.690 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0. 10-30-2020 20:28:15.694 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1. 10-30-2020 20:28:19.691 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0. 10-30-2020 20:28:24.695 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1. 10-30-2020 20:28:28.691 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0. 10-30-2020 20:28:29.193 +0000 INFO TcpOutputProc - Found currently active indexer. Connected to idx=172.19.0.4:9997, reuse=1. 10-30-2020 20:28:33.560 +0000 INFO TcpOutputProc - After randomization, current is first in the list. Swapping with last item 10-30-2020 20:28:33.661 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1. 10-30-2020 20:28:37.658 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0. 10-30-2020 20:28:42.561 +0000 INFO TcpOutputProc - After randomization, current is first in the list. Swapping with last item 10-30-2020 20:28:42.661 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1. 10-30-2020 20:28:46.659 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0. 10-30-2020 20:28:51.664 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1. 10-30-2020 20:28:55.661 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0. 10-30-2020 20:28:59.061 +0000 INFO TcpOutputProc - Found currently active indexer. Connected to idx=172.19.0.4:9997, reuse=1. 10-30-2020 20:29:00.662 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1. 10-30-2020 20:29:04.626 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0. 10-30-2020 20:29:09.530 +0000 INFO TcpOutputProc - After randomization, current is first in the list. Swapping with last item 10-30-2020 20:29:09.630 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1. ^C [root@uf1 splunkforwarder]#

So, I can see that both servers are taken into account but it always skips one and only uses the same other.

That is also what I see on both nodes; only 1 out of two in getting the tcp inputs.

So I got rid of the cloning, but this is also not what I want.

Is this what you expect?

Re: Load Balancing UF to 3rd third party receivers

nwuest — Sat, 31 Oct 2020 19:30:02 GMT

Hi @jknulst !

Your output does look like your Universal Forwarder is only sticking with one indexer in the the "default-auto-lb" group.
So now that we have the outputs.conf file solved, we need to verify a few other things in the Splunk Environment.

My next course of action would be to check the Splunk version on the 3rd party nodes and what your version is on your universal forwarder.

Can you also share a snippet from the metrics.log (from your universal forwarder AND see if you can get someone to look at the 3rd party indexer) in the same folder so we can see if there is a connection (Splunk-2-Splunk) error between the Universal Forwarder and the 3rd Party Indexers?

NOTE: The metrics.log should give us some good information as to why the connection is not succeeding with the "not-so-cooperative" 3rd party indexer.

On the Universal Forwarder (Grepping for the indexer that seems to not want to cooperate with the Universal Forwarder):
# tail -f /opt/splunkforwarder/var/log/splunk/metrics.log | grep "172.19.0.3"

On the 3rd Party Indexer (Grepping for the universal forwarder that seems to have connection issues):
# tail -f /opt/splunk/var/log/splunk/metrics.log | grep "ipaddress of splunk universal forwarder"

As always, Please let us know what you see from these commands so we can help troubleshoot further.

V/R,
nwuest

Re: Load Balancing UF to 3rd third party receivers

jknulst — Mon, 02 Nov 2020 13:55:58 GMT

@nwuest

You may have misunderstood, but with 3rd party receiver I really mean a 3rd party so not a Splunk receiver. It is in fact a distributed ETL tool that is receiving the data.

Does the UF also support load balancing as configured above to multiple addresses of a non-Splunk platform (with non-Cooked data)?

Re: Load Balancing UF to 3rd third party receivers

nwuest — Wed, 04 Nov 2020 04:36:20 GMT

Hi @jknulst

Thanks for the clarification.

Splunk is able to send data to third party systems Forward data to third-party systems

I’m more than sure that Splunk will be able to load balance as long as you add them to your outputs.conf file.

Be sure to update us on your progress with the 3rd party receivers.

V/R,
nwuest