topic Programmatically determine data processed by splunk forwarder file reader in Getting Data In

Programmatically determine data processed by splunk forwarder file reader

onelog2012 — Fri, 22 Jun 2012 18:15:09 GMT

How can we programmatically determine which lines of a log file have been successfully processed by a splunk forwarder log reader? With this information, we will be able to rollover and cleanup files quickly and know there is no data loss.

Re: Programmatically determine data processed by splunk forwarder file reader

Ayn — Fri, 22 Jun 2012 19:23:25 GMT

This should be of great interest to you: http://blogs.splunk.com/2011/01/02/did-i-miss-christmas-2/

Re: Programmatically determine data processed by splunk forwarder file reader

kristian_kolb — Sun, 24 Jun 2012 18:56:17 GMT

Ayn has a good point but you might also want to consider using indexer acknowledgement in combination with batch processing of the input files. With indexer acknowledgement, the indexer sends an acknowledgement when a piece of data has been written to disk in an index. Until then the forwarder keeps it in its output queue, and will resend it until it gets an acknowledgement. Pretty much like TCP network communications.

Batch reading of files (as opposed to monitoring) is a destructive method where a file is deleted after it has been processed (i.e. indexed).

More information on this can be found here:

Indexer Acknowledgement
http://docs.splunk.com/Documentation/Splunk/5.0/Deploy/Protectagainstlossofin-flightdata

Batch reading
http://docs.splunk.com/Documentation/Splunk/4.3.2/Data/Editinputs.conf

Hope this helps,

Kristian