How can I tell whether Splunk forwarder has read through monitored files

[sylim_splunk]

We have rsyslog writing files to numerous directories on Splunk heavy forwarders. In order to keep the logfiles from growing to unmanageable size and filling the disk on the forwarders, we would like to have rsyslog start a new file every hour. We need to be able to determine when it is safe to compress and/or remove the old files.

Cron jobs are no good if queues are blocked or splunkd not running for extended periods.

Splunk queries (search for last message in a file) are not good, since they would require authentication and would impose an unacceptable search load (there are thousands of files across 60 heavy forwarders).

We can't use batch inputs, as we can't tolerate delays of up to an hour ingesting data.

Logrotate causes mangled and dropped events when it runs, and the more often it runs the more damage it causes.

So... we need some way to be able to tell when Splunk has read all of a file being monitored, or a count of the number of lines read, or something we can use to know when it's safe to remove a syslog file that's no longer being written.

[sylim_splunk · Accepted solution]

Splunk command, "btprobe" can tell you how much it read which you can compare with actual file size. Moreover it doesn't require password either.

$ ./splunk cmd btprobe -d /home/fwd652/splunkforwarder/var/lib/splunk/fishbucket/splunk_private_db --file /home/fwd652/splunkforwarder/var/log/splunk/splunkd.log
Using logging configuration at /home/fwd652/splunkforwarder/etc/log-cmdline.cfg.
key=0xf002616871e8cce3 scrc=0x6d35eb146477b364 sptr=9881445 fcrc=0x4d8af36f7d891662 flen=0 mdtm=1510000676 wrtm=1510000679

In the result of "btprobe", find value for "sptr" = 9881445. This indicate how much splunk read the file.
Below is against a static file, splunkd.log.1, below shows "sptr == file size".

$ ./splunk cmd btprobe -d /home/fwd652/splunkforwarder/var/lib/splunk/fishbucket/splunk_private_db --file /home/fwd652/splunkforwarder/var/log/splunk/splunkd.log.1
Using logging configuration at /home/fwd652/splunkforwarder/etc/log-cmdline.cfg.
key=0xc1965e5752d40688 scrc=0x121343f558ae9a0f *sptr=25000066 fcrc=0x4d8af36f7d891662 flen=0 mdtm=1508580949 wrtm=1508580949*

$ ls -l /home/fwd652/splunkforwarder/var/log/splunk/splunkd.log.1
-rw------- 1 splunk splunk 25000066 Oct 21 06:15 ../var/log/splunk/splunkd.log.1