We try to monitor a single Logfile with a Splunk Universal Forwarder on a Windows Server 2008 R2 Server. In this Logfile, the newest Events always get posted at the top of the file.
If I use a Basic Setting like this:
index = app
sourcetype = System
recursive = false
whitelist = Filename.log
blacklist = otherFilename
It works fine first, but then it starts logging all Events over and over again. In the Splunkd.log i get following error:
03-24-2015 10:31:22.040 +0100 INFO WatchedFile - Will begin reading at offset=0 for file='D:\...forder\Filename.log'.
If I try the Option
followTail=true, it doesn't work anymore. It doesn't send anything to my Splunk indexer.
Does someone know this problem or is there a default solution? Unfortunately, I couldn't find a parameter to change the order of the logfile.
This is going to be a problem for Splunk, which expects the newest events to be at the end of the file.
Whenever Splunk sees that the beginning of a file has changed, it assumes that it is a new file and re-indexes the whole thing. This is what is happening to this file now. Using
crcSalt would turn off this behavior - BUT it will not make Splunk index the new events only.
I don't know of any Splunk settings which would properly configure an input like this. My only suggestion is this: write a script that periodically reviews the log and extracts only the new events and sends them to Splunk. Hopefully someone else has a better idea.
Or, fix the logging so that it writes to the end of the file.