About jknulst

jknulst · ‎11-02-2020

@nwuest You may have misunderstood, but with 3rd party receiver I really mean a 3rd party so not a Splunk receiver. It is in fact a distributed ETL tool that is receiving the data. Does the UF also support load balancing as configured above to multiple addresses of a non-Splunk platform (with non-Cooked data)?

jknulst · ‎10-30-2020

I followed your suggestion. The output is as follows: [root@uf1 splunkforwarder]# bin/splunk list forward-server Active forwards: 3rd_party_node_1:3535 3rd_party_node_2:3535 so1:9997 Configured but inactive forwards: None [root@uf1 splunkforwarder]# And the tail: [root@uf1 splunkforwarder]# tail -f /opt/splunkforwarder/var/log/splunk/splunkd.log | grep "TcpOutputProc" 10-30-2020 20:27:34.719 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0. 10-30-2020 20:27:39.613 +0000 INFO TcpOutputProc - After randomization, current is first in the list. Swapping with last item 10-30-2020 20:27:39.713 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1. 10-30-2020 20:27:43.720 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0. 10-30-2020 20:27:48.724 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1. 10-30-2020 20:27:52.722 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0. 10-30-2020 20:27:57.723 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1. 10-30-2020 20:27:59.335 +0000 INFO TcpOutputProc - Found currently active indexer. Connected to idx=172.19.0.4:9997, reuse=1. 10-30-2020 20:28:01.691 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0. 10-30-2020 20:28:06.719 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1. 10-30-2020 20:28:10.690 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0. 10-30-2020 20:28:15.694 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1. 10-30-2020 20:28:19.691 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0. 10-30-2020 20:28:24.695 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1. 10-30-2020 20:28:28.691 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0. 10-30-2020 20:28:29.193 +0000 INFO TcpOutputProc - Found currently active indexer. Connected to idx=172.19.0.4:9997, reuse=1. 10-30-2020 20:28:33.560 +0000 INFO TcpOutputProc - After randomization, current is first in the list. Swapping with last item 10-30-2020 20:28:33.661 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1. 10-30-2020 20:28:37.658 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0. 10-30-2020 20:28:42.561 +0000 INFO TcpOutputProc - After randomization, current is first in the list. Swapping with last item 10-30-2020 20:28:42.661 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1. 10-30-2020 20:28:46.659 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0. 10-30-2020 20:28:51.664 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1. 10-30-2020 20:28:55.661 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0. 10-30-2020 20:28:59.061 +0000 INFO TcpOutputProc - Found currently active indexer. Connected to idx=172.19.0.4:9997, reuse=1. 10-30-2020 20:29:00.662 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1. 10-30-2020 20:29:04.626 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.3:3535, pset=0, reuse=0. 10-30-2020 20:29:09.530 +0000 INFO TcpOutputProc - After randomization, current is first in the list. Swapping with last item 10-30-2020 20:29:09.630 +0000 INFO TcpOutputProc - Connected to idx=172.19.0.2:3535, pset=0, reuse=1. ^C [root@uf1 splunkforwarder]# So, I can see that both servers are taken into account but it always skips one and only uses the same other. That is also what I see on both nodes; only 1 out of two in getting the tcp inputs. So I got rid of the cloning, but this is also not what I want. Is this what you expect?

jknulst · ‎10-30-2020

@nwuest Thank you for your reponse and suggestion

jknulst · ‎10-21-2020

Hi, I have some troubles setting up the following topology. There is 1 UF which needs to forward unCooked raw data to a 3rd party receiver that is distributed and consists of 2 nodes. [indexAndForward] index = false [tcpout:splunk-searchhead-group] disabled = false server = so1:9997 [tcpout-server://so1:9997] [tcpout-server://3rd_party_node_1:3535] [tcpout-server://3rd_party_node_2:3535] [tcpout] defaultGroup = splunk-searchhead-group [tcpout:default-autolb-group] disabled = false server = 3rd_party_node_1:3535,3rd_party_node_2:3535 sendCookedData = false forceTimebasedAutoLB = true autoLBVolume = 2 autoLBFrequency = 5 maxQueueSize = auto indexAndForward = false blockOnCloning = true compressed = false dropClonedEventsOnQueueFull = 5 dropEventsOnQueueFull = -1 heartbeatFrequency = 30 maxFailuresPerInterval = 2 secsInFailureInterval = 1 maxConnectionsPerIndexer = 2 connectionTimeout = 20 readTimeout = 300 writeTimeout = 300 tcpSendBufSz = What happens in reality is that both 3rd_party_node_1 & 2 receive exactly the same data, it looks like data cloning in stead of load balancing. Is there anything off in this config or is load balancing not possible with 3rd party receivers? Thanks

Posts	5
Solutions	0
Karma Given	0
Karma Received	2
Member Since	‎10-21-2020

Online Status	Offline
Date Last Visited	‎11-02-2020 03:19 PM

Load Balancing UF to 3rd third party receivers

Re: Load Balancing UF to 3rd third party receiver...

Re: Load Balancing UF to 3rd third party receiver...

Re: Load Balancing UF to 3rd third party receiver...

Load Balancing UF to 3rd third party receivers