Getting Data In

How can I tell whether Splunk forwarder has read through monitored files

sylim_splunk
Splunk Employee
Splunk Employee

We have rsyslog writing files to numerous directories on Splunk heavy forwarders. In order to keep the logfiles from growing to unmanageable size and filling the disk on the forwarders, we would like to have rsyslog start a new file every hour. We need to be able to determine when it is safe to compress and/or remove the old files.

Cron jobs are no good if queues are blocked or splunkd not running for extended periods.

Splunk queries (search for last message in a file) are not good, since they would require authentication and would impose an unacceptable search load (there are thousands of files across 60 heavy forwarders).

We can't use batch inputs, as we can't tolerate delays of up to an hour ingesting data.

Logrotate causes mangled and dropped events when it runs, and the more often it runs the more damage it causes.

So... we need some way to be able to tell when Splunk has read all of a file being monitored, or a count of the number of lines read, or something we can use to know when it's safe to remove a syslog file that's no longer being written.

Tags (2)
0 Karma
1 Solution

sylim_splunk
Splunk Employee
Splunk Employee

Splunk command, "btprobe" can tell you how much it read which you can compare with actual file size. Moreover it doesn't require password either.

$ ./splunk cmd btprobe -d /home/fwd652/splunkforwarder/var/lib/splunk/fishbucket/splunk_private_db --file /home/fwd652/splunkforwarder/var/log/splunk/splunkd.log
Using logging configuration at /home/fwd652/splunkforwarder/etc/log-cmdline.cfg.
key=0xf002616871e8cce3 scrc=0x6d35eb146477b364 sptr=9881445 fcrc=0x4d8af36f7d891662 flen=0 mdtm=1510000676 wrtm=1510000679

In the result of "btprobe", find value for "sptr" = 9881445. This indicate how much splunk read the file.
Below is against a static file, splunkd.log.1, below shows "sptr == file size".

$ ./splunk cmd btprobe -d /home/fwd652/splunkforwarder/var/lib/splunk/fishbucket/splunk_private_db --file /home/fwd652/splunkforwarder/var/log/splunk/splunkd.log.1
Using logging configuration at /home/fwd652/splunkforwarder/etc/log-cmdline.cfg.
key=0xc1965e5752d40688 scrc=0x121343f558ae9a0f *sptr=25000066 fcrc=0x4d8af36f7d891662 flen=0 mdtm=1508580949 wrtm=1508580949*

$ ls -l /home/fwd652/splunkforwarder/var/log/splunk/splunkd.log.1
-rw------- 1 splunk splunk 25000066 Oct 21 06:15 ../var/log/splunk/splunkd.log.1

View solution in original post

sylim_splunk
Splunk Employee
Splunk Employee

Splunk command, "btprobe" can tell you how much it read which you can compare with actual file size. Moreover it doesn't require password either.

$ ./splunk cmd btprobe -d /home/fwd652/splunkforwarder/var/lib/splunk/fishbucket/splunk_private_db --file /home/fwd652/splunkforwarder/var/log/splunk/splunkd.log
Using logging configuration at /home/fwd652/splunkforwarder/etc/log-cmdline.cfg.
key=0xf002616871e8cce3 scrc=0x6d35eb146477b364 sptr=9881445 fcrc=0x4d8af36f7d891662 flen=0 mdtm=1510000676 wrtm=1510000679

In the result of "btprobe", find value for "sptr" = 9881445. This indicate how much splunk read the file.
Below is against a static file, splunkd.log.1, below shows "sptr == file size".

$ ./splunk cmd btprobe -d /home/fwd652/splunkforwarder/var/lib/splunk/fishbucket/splunk_private_db --file /home/fwd652/splunkforwarder/var/log/splunk/splunkd.log.1
Using logging configuration at /home/fwd652/splunkforwarder/etc/log-cmdline.cfg.
key=0xc1965e5752d40688 scrc=0x121343f558ae9a0f *sptr=25000066 fcrc=0x4d8af36f7d891662 flen=0 mdtm=1508580949 wrtm=1508580949*

$ ls -l /home/fwd652/splunkforwarder/var/log/splunk/splunkd.log.1
-rw------- 1 splunk splunk 25000066 Oct 21 06:15 ../var/log/splunk/splunkd.log.1

Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...