Cloned Destinations: blockOnCloning

[ak_sky]

Could someone confirm the expected outcome for the following settings:

outputs.conf

[tcpout:group1]
server = 192.168.10.1, 192.168.10.2
sendCookedData = true
blockOnCloning = true
# Default block on cloning setting

[tcpout:group2]
server = 192.168.20.1, 192.168.20.2
sendCookedData = true
blockOnCloning = false
# Alternative blockOnCloning setting

inputs.conf
[monitor:////var/log/syslog]
index=linux
sourcetype=syslog
_TCP_ROUTING = group1,group2

1) If group1 is unavailable, but group2 is available

2) If group1 is available, but group2 is unavailable

3) If group 1 and group2 are both unavailable

I have read the description in the spec file but am not sure how to correctly interpret it:

blockOnCloning = <boolean>
* Whether or not the TcpOutputProcessor should wait until at least one
  of the cloned output groups receives events before attempting to send
  more events.
* If set to "true", the TcpOutputProcessor blocks until at least one of the
  cloned groups receives events. It does not drop events when all the
  cloned groups are down.
* If set to "false", the TcpOutputProcessor drops events when all the
  cloned groups are down and all queues for the cloned groups are full.
  When at least one of the cloned groups is up and queues are not full,
  the events are not dropped.
* Default: true

My end goal is to configure such that event if one TCPOUT group is unavailable logging should continue to the other unaffected, even if that means the unavailable group will miss receving some events.

Thanks

[woodcock]

I could be wrong but I believe that the documentation is in error here because as written, the feature is useless.  If my memory is correct, it works like this:

1) If group1 is unavailable, but group2 is available
blockOnCloning=false: Send events to whomever is receiving (e.g. group 1).
blockOnCloning=true: Only send events if everybody is receiving (e.g. nobody)

2) If group1 is available, but group2 is unavailable
blockOnCloning=false: Send events to whomever is receiving (e.g. group 2).
blockOnCloning=true: Only send events if everybody is receiving (e.g. nobody)

3) If group 1 and group2 are both unavailable

blockOnCloning=false: Send events to whomever is receiving (e.g. nobody).
blockOnCloning=true: Only send events if everybody is receiving (e.g. nobody)

[hrawat]

Yes it's really useless.

Use following enhanced settings.

dropEventsOnQueueFull=0ms 

dropClonedEventsOnQueueFull=0ms

 Let's say I have two groups primary and secondary. No events should be dropped for primary but ok to drop for secondary.

[tcpout:primary]
dropEventsOnQueueFull=-1
dropClonedEventsOnQueueFull=-1

[tcpout:secondary]
dropEventsOnQueueFull=0ms
dropClonedEventsOnQueueFull=0ms

If you are expecting that data must flow if one goes down(totally unreachable)

[tcpout:destinationA]
dropEventsOnQueueFull=<how many seconds to wait before dropping starts>
dropClonedEventsOnQueueFull=<how many seconds to wait before dropping starts>

[tcpout:destinationB]
dropEventsOnQueueFull=<how many seconds to wait before dropping starts>
dropClonedEventsOnQueueFull=<how many seconds to wait before dropping starts>

Globally

[tcpout]
dropEventsOnQueueFull=<how many seconds to wait before dropping starts>
dropClonedEventsOnQueueFull=<how many seconds to wait before dropping starts>

[PickleRick]

What is important here and is easy to miss is that you need to set

dropEventsOnQueueFull=0ms
dropClonedEventsOnQueueFull=0ms

Not just plain "0", it must be "0ms"! 

[hrawat]

---

@PickleRick wrote:

What is important here and is easy to miss is that you need to set

dropEventsOnQueueFull=0ms
dropClonedEventsOnQueueFull=0ms

Not just plain "0", it must be "0ms"! 

---

Historically 0 and -1 were same. Don't drop the data. In order to not break existing configs, we introduced [ms|s|m]. So that drop `0ms` or `0s` or `0m` will drop data instantaneously. 

[PickleRick]

I suspected something along those lines 🙂

It's just that it's a bit counterintuitive since usually in configs you have some default units which can be omitted. This case is different so I pointed it out because the behaviour can be a bit unexpected.

[SDSplQuestion]

Hi Woodcock,

May I please double check the nature of this setting as it stands today, say if I have the below:

[tcpout]
defaultGroup = group1, group2
blockOnCloning = [0]

[tcpout:group1]
server = server1:9997
blockOnCloning = [1]

[tcpout:group2]
server = server2:9997
blockOnCloning = [2]

Would the outcomes be as follows, I want to check if it being in the main tcpout supersedes the separate groups but also want to make sure if one side collapses, the other is fine.

ID	0	1	2	Outcome if Server 1 collapses	Outcome if Server 2 collapses
1	true	true	true	Results stopped for both	Results stopped for both
2	true	true	false	Results stopped for both	Results continue for 1
3	true	false	false	Results stopped for both	Results stopped for both
4	false	false	false	Results continue for 2	Results continue for 1
5	false	false	true	Results continue for 2	Results stopped for both
6	false	true	true	Results stopped for both	Results stopped for both