How to edit my universal forwarder's monitor configuration for a single log file to prevent indexing events over and over again?

[lukas_loder]

Hello,

We try to monitor a single Logfile with a Splunk Universal Forwarder on a Windows Server 2008 R2 Server. In this Logfile, the newest Events always get posted at the top of the file.

If I use a Basic Setting like this:

[monitor://D:\...\folder\]
index = app
sourcetype = System
recursive = false
whitelist = Filename.log
blacklist = otherFilename
disabled=0

It works fine first, but then it starts logging all Events over and over again. In the Splunkd.log i get following error:

03-24-2015 10:31:22.040 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='D:\...forder\Filename.log'.

If I try the Option followTail=1 or followTail=true, it doesn't work anymore. It doesn't send anything to my Splunk indexer.

Does someone know this problem or is there a default solution? Unfortunately, I couldn't find a parameter to change the order of the logfile.

Thanks!

[lguinn2]

This is going to be a problem for Splunk, which expects the newest events to be at the end of the file.

Whenever Splunk sees that the beginning of a file has changed, it assumes that it is a new file and re-indexes the whole thing. This is what is happening to this file now. Using crcSalt would turn off this behavior - BUT it will not make Splunk index the new events only.

I don't know of any Splunk settings which would properly configure an input like this. My only suggestion is this: write a script that periodically reviews the log and extracts only the new events and sends them to Splunk. Hopefully someone else has a better idea.

Or, fix the logging so that it writes to the end of the file.

[satishsdange]

Are you using crcSalt in props.conf?

[lukas_loder]

No, I'm not using a props.conf for this at all. How would it work with crcSalt?