Light Forwarder sent log messages twice

[skippylou]

So I have a light forwarder (4.1.3) reading snort logs with the following configuration in /opt/splunk/etc/apps/search/local/inputs.conf:

[monitor:///var/log/by2/output/alert_full*]
disabled = false
followTail = 1
sourcetype = snort
source = snort
index = security

There is currently only a single file in that directory that matches, which is alert_full.log. The LWF has been sending in events as they arrive perfectly for 4 or 5 days.

The server this LWF is running on was rebooted the other day and when it came back up, I noticed on my splunk indexer in splunk web that the whole alert_full.log file on the LWF had been completely re-read and every event in it from when I started days back was indexed again. This time all the events did not have the correct timestamp, but all had a splunk timestamp of when it was indexed (not that this really matters much as they shouldn't haven been re-indexed anyhow).

I thought I remember restarting splunk on the LWF previously and this not happening, but maybe I had not yet.

Any reason this would have happened? Where should I look to see where the LWF thinks it left off in a file it is monitoring (perhaps this file got removed/corrupted/etc.)?

Thanks,

scott

[skippylou]

Right after restart, I don't see much out of the ordinary, it has the Tailing Processor for that monitor setup i have in inputs and the only errors I see are these, not sure what they mean:

09-14-2010 09:40:32.769 ERROR pipeline - Empty pipeline (no processors): scheduler, exiting pipeline
09-14-2010 09:40:32.769 ERROR IndexProcessor - received event for unconfigured/disabled index='_audit' with source='source::audittrail' host='host::myhostname' sourcetype='sourcetype::audittrail'

A third restart did not cause the same issue...

[jrodman · Accepted solution]

Sounds wrong on our part. It's possible that the btree file got sad somehow, but that would be a new one. It's also possible that the alert_full.log file got subtly changed, eg a byte change in the first 256 bytes.

Both those are a bit unlikely but if you can see that they happened would explain it for sure.

Personally I'd dump the followTail setting.

[wrangler2x]

I have a splunk system on campus here at UC Irvine that is using a portion of my license pool. It is monitoring the Apache log on a Windows system. They get about 35 megabytes a day of new data in it, but over the last week they have been logging over 7 gigs of data a day, violating the license (100 megs). The size of the Apache log file is currently at 7.6 gig). Is this still a bug (they are on 4.3) in followTail? What is the alternative to doing that?

[skippylou]

just noticed this issue is actually in the release notes for 4.1.5:

monitor inputs using the followTail setting sometimes will index some older events or all events from log files which are updated when not intended. (SPL-23555)

[jrodman]

followTail is only useful the very first time you index a given file / directory. Beyond that, it has a few warts. I'm not sure if they are affecting you.

[skippylou]

Nothing looks out of the ordinary in the alert_full.log that I can tell (such as out of order alerts, etc.).

Just out of curiosity, why would you dump the followTail setting?

[Genti]

splunkd.log might be a good place to start. check what splunkd says right after you restarted. If you restart your LWF again, does it reindex everything for the 3rd time? It seems like the CRC is changing, but i cant seem to understand why with the given info...