Archive

Does Universal Forwarder connection Loss cause data loss ?

DavidHourani
Super Champion

Hello,

I need help on understanding something.

If I have a folder being monitored by a universal forwarder and I lose the connectivity between the universal forwarder and the indexer. Does that cause the loss of all the data that arrives on the monitored folder during connection loss time ?

Do i need to use Ack to ensure that universal ?

In which case do I need to have Ack enabled ? It's only when i have streams/scripts arriving on the forwarder without storing the results on disk right ?

Regards,
David

0 Karma

DavidHourani
Super Champion

Thank you for your answers, the cases stated in your link are for scripted inputs where data isn't written to disk. In my case I have the data on files already and the files are configured with log rotation so if the connectivity is lost, the data will remain in those files, I don't see the use of ack+queues in this case. Doesn't the universal forwarder set some sort of pointer to follow which data in the file has been send and which hasn't ?

0 Karma

somesoni2
Revered Legend

nilbak1
Communicator

Hi @DavidHourani
I am also having same query if data is travelling over tcp port, and don't use data acknowledgement setting then there will be data loss ?

0 Karma

DavidHourani
Super Champion

Hi Nilbak, did some digging around and here's what I found : tcp will guarantee that your data crosses across the network but doesn't guarantee the reception at application level, so on layer 4 you will receive data but data might be missing at app layer. Ack adds this validation of data at the reception but might cause duplicates and slows down perf, I would avoid it if it's not a strict requirement.

0 Karma

nilbak1
Communicator

Thanks DavaidHourani..
So, that means we can go ahead and remove the acknowledgement settings in our endpoints since its not a strict requirement and data is transmitting over tcp, only concern was here data loss.

ddrillic
Ultra Champion

@nilbak1, the acknowledgement settings are part of making this open platform product an enterprise solution. By stripping it, you make it a bit less of an enterprise solution. So, the question you should ask is - what motivation do I have not to have the acknowledgement? Only if you have a valid reason I would strip it.

0 Karma

DavidHourani
Super Champion

Sorry @ddrillic your answer is wrong. Please read doc carefully before activating acknowledgement.

0 Karma

ddrillic
Ultra Champion

Oh ok, I see what you are saying -

-- For instance, assume the indexer receives a data block, parses it, and writes it to the file system. It then generates the acknowledgment. However, on the round-trip to the forwarder, the network goes down, so the forwarder never receives the acknowledgment. When the network comes back up, the forwarder then resends the data block, which the indexer will parse and write as if it were new data.

But you know, it's all about probabilities - with all the endless Splunk outages that I experienced in the past couple of years, none of them involved the network going down. Forwarders and indexers, in my world, went down hundreds of times, involving huge amount of data that got lost in transit. Acknowledgment would have made of data reliability and integrity much stronger. That's my experience ; -)

DavidHourani
Super Champion

Yeah, as I wrote earlier, it's all about your data policy. Cost of Acks is lower performance, delays in indexing and duplications, to be avoided as long as it's not a strict requirement. If in your case you've tested and you're not experiencing the above then that's great, I've seen a lot of clients with those issues that's why it should be considered carefully.

ddrillic
Ultra Champion

@DavidHourani, why would it cause duplicates? Btw, the Splunk Architect classes emphasize that acknowledgement should be used.

0 Karma

DavidHourani
Super Champion

Cool, if your architect requires it then you will have to go for it, but as I said it has to be a strict requirement and not just some guy saying go for it, I personally would advise against it if it's not a defined in your corporate data retention policy.

Here's the info about the possible duplicates: http://docs.splunk.com/Documentation/Splunk/7.2.1/Forwarding/Protectagainstlossofin-flightdata#The_p...
And also expect perf problems and delays in log reception. Let me know if you need further details.

ChrisG
Splunk Employee
Splunk Employee
0 Karma

DavidHourani
Super Champion

Thank you for your answers, the cases stated in your link are for scripted inputs where data isn't written to disk. In my case I have the data on files already and the files are configured with log rotation so if the connectivity is lost, the data will remain in those files, I don't see the use of ack+queues in this case. Doesn't the universal forwarder set some sort of pointer to follow which data in the file has been send and which hasn't ?

0 Karma

somesoni2
Revered Legend

The data acknowledgement works same way regardless of the type of input. Splunk forwarder does keep track of which data has been sent, but without the useAck, it doesn't know which data has been received by indexer, and can cause data loss if UF/Indexer is down/no connection.

0 Karma

DavidHourani
Super Champion

But forwarding is tcp based, so how can we have data loss for file based forwarding if we're sending one tcp window size at a time ?

0 Karma

FrankVl
Ultra Champion

When the UF completely looses connection: you're right, it will just stop sending (and reading once queues fill up) and continue where it left off once connection is restored.
But when the UF is able to send out the event, but the indexer is unable to actually index it (e.g. splunk crashes or so during processing) you could still loose data when ACK is not enabled.