Hello,
We have a question regarding a specific use case of data forwarding, we would like to know if there is a risk with the situation.
Let's say we have two Splunk platforms, one has a set of ...
See more...
Hello,
We have a question regarding a specific use case of data forwarding, we would like to know if there is a risk with the situation.
Let's say we have two Splunk platforms, one has a set of indexers (indexer1) that stores data for a specific use case, the other platform has also its indexer (indexer2) for another use case. At some point, these 2 platforms will have to collect data from the same machines, not necessarily the same data, but it could be.
So there is a forwarder on a remote machine, let's say we have to collect the same file and forward it to both platforms. We are using the data cloning technique, with for example:
In inputs.conf
[monitor://file_path]
index = …
sourcetype = ...
_TCP_ROUTING=indexer1,indexer2
Or with outputs.conf
[tcpout]
defaultGroup=indexer1,indexer2
Or by using props/transforms to alter the routing of the events on an intermediate Heavy Forwarder, whatever (please, do tell if there is a significant difference with any of these methods for this situation).
Now the question is: In general, is there a risk that one of the two platforms will stop receiving events if the other is down, depending on the configuration of the indexers/forwarders ?
We have heard of the concept of indexer acknowledgment, we are not sure if it can have any impact on this situation. For example, if the group indexer1 is configured with acknowledgement enabled, is there any risk that the group indexer2 won't receive data when indexer1 is not acknowledging the reception ?
This topic is a little bit confusing for us, we have heard claims that the data forwarding could be blocked if another platform needs to receive the data and one of the indexer is down, but it doesn't seem right. We just want to clarify, with the set up described above, if there would be any issue.
Thank you very much for your help
Update:
We learned about the parameters in outputs.conf to consider when configuring the behavior of the queues:
- dropEventsOnQueueFull =
- dropClonedEventsOnQueueFull =
- blockOnCloning =
It is indeed possible to block the data collect of a splunk instance (in a data cloning configuration) when not paying attention to these parameters. In the little test we did, the default value of dropClonedEventsOnQueueFull made it so that the data collect didn't block. However we have to watch out for dropEventsOnQueueFull as well, which can cause data forwarding issues when a splunk instance is unavailable (with default value) => But it also depends on whether you accept the loss of data or not in your deployment.
Very interesting parameters to know about.