Getting Data In

How to edit my universal forwarder's monitor configuration for a single log file to prevent indexing events over and over again?



We try to monitor a single Logfile with a Splunk Universal Forwarder on a Windows Server 2008 R2 Server. In this Logfile, the newest Events always get posted at the top of the file.

If I use a Basic Setting like this:

index = app
sourcetype = System
recursive = false
whitelist = Filename.log
blacklist = otherFilename

It works fine first, but then it starts logging all Events over and over again. In the Splunkd.log i get following error:

03-24-2015 10:31:22.040 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='D:\...forder\Filename.log'.

If I try the Option followTail=1 or followTail=true, it doesn't work anymore. It doesn't send anything to my Splunk indexer.

Does someone know this problem or is there a default solution? Unfortunately, I couldn't find a parameter to change the order of the logfile.


0 Karma


This is going to be a problem for Splunk, which expects the newest events to be at the end of the file.

Whenever Splunk sees that the beginning of a file has changed, it assumes that it is a new file and re-indexes the whole thing. This is what is happening to this file now. Using crcSalt would turn off this behavior - BUT it will not make Splunk index the new events only.

I don't know of any Splunk settings which would properly configure an input like this. My only suggestion is this: write a script that periodically reviews the log and extracts only the new events and sends them to Splunk. Hopefully someone else has a better idea.

Or, fix the logging so that it writes to the end of the file.


Are you using crcSalt in props.conf?

0 Karma


No, I'm not using a props.conf for this at all. How would it work with crcSalt?

0 Karma
Get Updates on the Splunk Community!

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Using the Splunk Threat Research Team’s Latest Security Content

REGISTER HERE Tech Talk | Security Edition Did you know the Splunk Threat Research Team regularly releases ...