topic Re: How do forwarders handle rolling logs when an indexer is down? in Getting Data In

How do forwarders handle rolling logs when an indexer is down?

andrewtrobec — Tue, 22 Sep 2020 16:21:08 GMT

Hello,

I would like to know how forwarders handle rolling logs when their target indexers become unavailable. Here is a simple scenario:

My application creates a log "application.log"
At midnight, "application.log" gets rolled to "application.backup" and a new "application.log" gets created

Assuming my indexer goes down at 11pm and gets restored at 1am the following day, there is 1 hour of log data that will get rolled to "application.backup" and 1 hour of data that is contained in the new "application.log" when the indexer gets restored.

My question relating to the above scenario: will the forwarder keep track of the hour's worth of data that was rolled to "application.backup" as well as the hour's worth of data that is written to "application.log" and send it to the indexer once it becomes available?

Thank you!

Andrew

Re: How do forwarders handle rolling logs when an indexer is down?

richgalloway — Tue, 22 Sep 2020 17:07:50 GMT

Forwarders buffer their data while they wait for an indexer to become available. If the data involved does not exceed the forwarder's queue (as set by queueSize and persistentQueueSize in inputs.conf and maxSize in server.conf) then no data will be lost.

Re: How do forwarders handle rolling logs when an indexer is down?

andrewtrobec — Wed, 23 Sep 2020 06:20:27 GMT

Thanks Rich!