Load Balancing UF to 3rd third party receivers

[jknulst]

Hi,

I have some troubles setting up the following topology. There is 1 UF which needs to forward unCooked raw data to a 3rd party receiver that is distributed and consists of 2 nodes.

[indexAndForward]
index = false

[tcpout:splunk-searchhead-group]
disabled = false
server = so1:9997

[tcpout-server://so1:9997]
[tcpout-server://3rd_party_node_1:3535]
[tcpout-server://3rd_party_node_2:3535]

[tcpout]
defaultGroup = splunk-searchhead-group

[tcpout:default-autolb-group]
disabled = false
server = 3rd_party_node_1:3535,3rd_party_node_2:3535
sendCookedData = false
forceTimebasedAutoLB = true
autoLBVolume = 2
autoLBFrequency = 5
maxQueueSize = auto
indexAndForward = false
blockOnCloning = true
compressed = false
dropClonedEventsOnQueueFull = 5
dropEventsOnQueueFull = -1
heartbeatFrequency = 30
maxFailuresPerInterval = 2
secsInFailureInterval = 1
maxConnectionsPerIndexer = 2
connectionTimeout = 20
readTimeout = 300
writeTimeout = 300
tcpSendBufSz =

What happens in reality is that both 3rd_party_node_1 & 2 receive exactly the same data, it looks like data cloning in stead of load balancing.

Is there anything off in this config or is load balancing not possible with 3rd party receivers?

Thanks

[jknulst]

I followed your suggestion.

The output is as follows:

[root@uf1 splunkforwarder]# bin/splunk list forward-server
Active forwards:
	3rd_party_node_1:3535
	3rd_party_node_2:3535
	so1:9997
Configured but inactive forwards:
	None
[root@uf1 splunkforwarder]# 

And the tail:

[root@uf1 splunkforwarder]# tail -f /opt/splunkforwarder/var/log/splunk/splunkd.log | grep "TcpOutputProc"
10-30-2020 20:27:34.719 +0000 INFO  TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0.
10-30-2020 20:27:39.613 +0000 INFO  TcpOutputProc - After randomization, current is first in the list. Swapping with last item
10-30-2020 20:27:39.713 +0000 INFO  TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1.
10-30-2020 20:27:43.720 +0000 INFO  TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0.
10-30-2020 20:27:48.724 +0000 INFO  TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1.
10-30-2020 20:27:52.722 +0000 INFO  TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0.
10-30-2020 20:27:57.723 +0000 INFO  TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1.
10-30-2020 20:27:59.335 +0000 INFO  TcpOutputProc - Found currently active indexer. Connected to idx=172.19.0.4:9997, reuse=1.
10-30-2020 20:28:01.691 +0000 INFO  TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0.
10-30-2020 20:28:06.719 +0000 INFO  TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1.
10-30-2020 20:28:10.690 +0000 INFO  TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0.
10-30-2020 20:28:15.694 +0000 INFO  TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1.
10-30-2020 20:28:19.691 +0000 INFO  TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0.
10-30-2020 20:28:24.695 +0000 INFO  TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1.
10-30-2020 20:28:28.691 +0000 INFO  TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0.
10-30-2020 20:28:29.193 +0000 INFO  TcpOutputProc - Found currently active indexer. Connected to idx=172.19.0.4:9997, reuse=1.
10-30-2020 20:28:33.560 +0000 INFO  TcpOutputProc - After randomization, current is first in the list. Swapping with last item
10-30-2020 20:28:33.661 +0000 INFO  TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1.
10-30-2020 20:28:37.658 +0000 INFO  TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0.
10-30-2020 20:28:42.561 +0000 INFO  TcpOutputProc - After randomization, current is first in the list. Swapping with last item
10-30-2020 20:28:42.661 +0000 INFO  TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1.
10-30-2020 20:28:46.659 +0000 INFO  TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0.
10-30-2020 20:28:51.664 +0000 INFO  TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1.
10-30-2020 20:28:55.661 +0000 INFO  TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0.
10-30-2020 20:28:59.061 +0000 INFO  TcpOutputProc - Found currently active indexer. Connected to idx=172.19.0.4:9997, reuse=1.
10-30-2020 20:29:00.662 +0000 INFO  TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1.
10-30-2020 20:29:04.626 +0000 INFO  TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0.
10-30-2020 20:29:09.530 +0000 INFO  TcpOutputProc - After randomization, current is first in the list. Swapping with last item
10-30-2020 20:29:09.630 +0000 INFO  TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1.
^C
[root@uf1 splunkforwarder]# 

So, I can see that both servers are taken into account but it always skips one and only uses the same other.

That is also what I see on both nodes; only 1 out of two in getting the tcp inputs.

So I got rid of the cloning, but this is also not what I want.

Is this what you expect?

[nwuest]

Hi @jknulst !

Your output does look like your Universal Forwarder  is only sticking with one indexer in the the "default-auto-lb" group.
So now that we have the outputs.conf file solved, we need to verify a few other things in the Splunk Environment. 

My next course of action would be to check the Splunk version on the 3rd party nodes and what your version is on your universal forwarder.

• Can you also share a snippet from the metrics.log (from your universal forwarder AND see if you can get someone to look at the 3rd party indexer) in the same folder so we can see if there is a connection (Splunk-2-Splunk) error between the Universal Forwarder and the 3rd Party Indexers?
  
  NOTE: The metrics.log should give us some good information as to why the connection is not succeeding with the "not-so-cooperative" 3rd party indexer.
  
  

On the Universal Forwarder (Grepping for the indexer that seems to not want to cooperate with the Universal Forwarder):
# tail -f /opt/splunkforwarder/var/log/splunk/metrics.log | grep "172.19.0.3"

On the 3rd Party Indexer (Grepping for the universal forwarder that seems to have connection issues):
# tail -f /opt/splunk/var/log/splunk/metrics.log | grep "ipaddress of splunk universal forwarder"

As always, Please let us know what you see from these commands so we can help troubleshoot further.

V/R,
nwuest

[jknulst]

@nwuest 

You may have misunderstood, but with 3rd party receiver I really mean a 3rd party so not a Splunk receiver. It is in fact a distributed ETL tool that is receiving the data.

Does the UF also support load balancing as configured above to multiple addresses of a non-Splunk platform (with non-Cooked data)?

[nwuest]

Hi @jknulst 

Thanks for the clarification. 

Splunk is able to send data to third party systems Forward data to third-party systems 

I’m more than sure that Splunk will be able to load balance as long as you add them to your outputs.conf file. 

Be sure to update us on your progress with the 3rd party receivers. 

V/R,
nwuest

[nwuest]

Hi @jknulst,

Your configuration looks good to me but needs one small tweak.

In your [tcpout] stanza you have the following:

[tcpout]
defaultGroup = splunk-searchhead-group

I believe you need to add the other tcpout:group you have defined in your outputs.conf file so that the Universal Forwarder begins the "load-balancing" between the two 3rd-party indexers/nodes

[tcpout]
defaultGroup = splunk-searchhead-group,default-autolb-group

Once you have set this, give your Splunk Universal Forwarder a reboot for the new configurations to take affect. 
Your Splunk Universal Forwarder should now switch between each 3rd-party indexer/node every 30 seconds.

Note: If your box you have installed on is a linux variant, you might be able to run this command and see the switch happen in real-time:
# watch -d -n 0 "/opt/splunkforwarder/bin/splunk list forward-server"
You will be prompted for your Splunk U.F.'s username and password. Once entered, it should rerun that command and you will hopefully see the switchover in real-time.

If you don't have the "watch" command you can always tail the /opt/splunkforwarder/var/log/splunk/splunkd.log for the following command to see your Universal Forwarder switch nodes/indexers.
# tail -f /opt/splunkforwarder/var/log/splunk/splunkd.log | grep "TcpOutputProc"

If you are currently running on a windows box and have access to powershell you can use the following command to see the output of splunkd.log
(Open powershell in admin mode)
# Get-Content -Path "C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log" -Wait
It will chunk through the file until it gets to the end but

Please let us know if this solves your current challenge!

V/R,
nwuest

[jknulst]

@nwuest Thank you for your reponse and suggestion