Getting Data In

How to edit my universal forwarder's monitor configuration for a single log file to prevent indexing events over and over again?



We try to monitor a single Logfile with a Splunk Universal Forwarder on a Windows Server 2008 R2 Server. In this Logfile, the newest Events always get posted at the top of the file.

If I use a Basic Setting like this:

index = app
sourcetype = System
recursive = false
whitelist = Filename.log
blacklist = otherFilename

It works fine first, but then it starts logging all Events over and over again. In the Splunkd.log i get following error:

03-24-2015 10:31:22.040 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='D:\...forder\Filename.log'.

If I try the Option followTail=1 or followTail=true, it doesn't work anymore. It doesn't send anything to my Splunk indexer.

Does someone know this problem or is there a default solution? Unfortunately, I couldn't find a parameter to change the order of the logfile.


0 Karma


This is going to be a problem for Splunk, which expects the newest events to be at the end of the file.

Whenever Splunk sees that the beginning of a file has changed, it assumes that it is a new file and re-indexes the whole thing. This is what is happening to this file now. Using crcSalt would turn off this behavior - BUT it will not make Splunk index the new events only.

I don't know of any Splunk settings which would properly configure an input like this. My only suggestion is this: write a script that periodically reviews the log and extracts only the new events and sends them to Splunk. Hopefully someone else has a better idea.

Or, fix the logging so that it writes to the end of the file.


Are you using crcSalt in props.conf?

0 Karma


No, I'm not using a props.conf for this at all. How would it work with crcSalt?

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Customer Survey!

If you use Splunk Observability Cloud, we invite you to share your valuable insights with us through a brief ...

Happy CX Day, Splunk Community!

Happy CX Day, Splunk Community! CX stands for Customer Experience, and today, October 3rd, is CX Day — a ...

.conf23 | Get Your Cybersecurity Defense Analyst Certification in Vegas

We’re excited to announce a new Splunk certification exam being released at .conf23! If you’re going to Las ...