Programmatically determine data processed by splunk forwarder file reader

[onelog2012]

How can we programmatically determine which lines of a log file have been successfully processed by a splunk forwarder log reader? With this information, we will be able to rollover and cleanup files quickly and know there is no data loss.

[kristian_kolb]

Ayn has a good point but you might also want to consider using indexer acknowledgement in combination with batch processing of the input files. With indexer acknowledgement, the indexer sends an acknowledgement when a piece of data has been written to disk in an index. Until then the forwarder keeps it in its output queue, and will resend it until it gets an acknowledgement. Pretty much like TCP network communications.

Batch reading of files (as opposed to monitoring) is a destructive method where a file is deleted after it has been processed (i.e. indexed).

More information on this can be found here:

Indexer Acknowledgement
http://docs.splunk.com/Documentation/Splunk/5.0/Deploy/Protectagainstlossofin-flightdata

Batch reading
http://docs.splunk.com/Documentation/Splunk/4.3.2/Data/Editinputs.conf

Hope this helps,

Kristian

[Ayn]

This should be of great interest to you: http://blogs.splunk.com/2011/01/02/did-i-miss-christmas-2/