topic Re: how often does the forwarder send data to indexer in Getting Data In

how often does the forwarder send data to indexer

mgh4 — Thu, 17 Jan 2013 16:58:18 GMT

We are forwarding logs and the question has come up as to how often the forwarder sends data and is there a way to flush the data when you stop the forwarder?

I looked in etc/system/README/outputs.conf.spec but didn't find anything relating to this interval.

Thanks

Re: how often does the forwarder send data to indexer

Ayn — Thu, 17 Jan 2013 17:03:16 GMT

The forwarder sends data immediately when it becomes available. There is no certain interval that it waits or anything like that.

Re: how often does the forwarder send data to indexer

mgh4 — Thu, 17 Jan 2013 17:08:51 GMT

Ok, but since it can send in blocks and set source type it must be waiting at least for an end of line, or for a file to not change over some time interval. Like, send changes if file has not update in 1 second.

Re: how often does the forwarder send data to indexer

Ayn — Thu, 17 Jan 2013 17:15:35 GMT

To be honest I don't know the exact behaviour here but Universal Forwarders do not perform any parsing or line breaking so they pretty much look for new data and pass it along as soon as they see it. It won't wait for a file to stop being updated.

Re: how often does the forwarder send data to indexer

yannK — Thu, 17 Jan 2013 17:37:00 GMT

while tailing a file, the forwarder consider that a line is complete if an open file has not be touched for up to 3 seconds, or immediately when the lock of the file is released.

However in particular cases, if you have an application writing with a buffer or irregularly, that can cause the beginning of the line to be written, pause, then write the end, you can experience a broken event. ( turned in to 2 events)

Re: how often does the forwarder send data to indexer

mgh4 — Thu, 17 Jan 2013 18:44:13 GMT

thank you. that makes sense.