Getting Data In

How to edit my universal forwarder's monitor configuration for a single log file to prevent indexing events over and over again?



We try to monitor a single Logfile with a Splunk Universal Forwarder on a Windows Server 2008 R2 Server. In this Logfile, the newest Events always get posted at the top of the file.

If I use a Basic Setting like this:

index = app
sourcetype = System
recursive = false
whitelist = Filename.log
blacklist = otherFilename

It works fine first, but then it starts logging all Events over and over again. In the Splunkd.log i get following error:

03-24-2015 10:31:22.040 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='D:\...forder\Filename.log'.

If I try the Option followTail=1 or followTail=true, it doesn't work anymore. It doesn't send anything to my Splunk indexer.

Does someone know this problem or is there a default solution? Unfortunately, I couldn't find a parameter to change the order of the logfile.


0 Karma


This is going to be a problem for Splunk, which expects the newest events to be at the end of the file.

Whenever Splunk sees that the beginning of a file has changed, it assumes that it is a new file and re-indexes the whole thing. This is what is happening to this file now. Using crcSalt would turn off this behavior - BUT it will not make Splunk index the new events only.

I don't know of any Splunk settings which would properly configure an input like this. My only suggestion is this: write a script that periodically reviews the log and extracts only the new events and sends them to Splunk. Hopefully someone else has a better idea.

Or, fix the logging so that it writes to the end of the file.


Are you using crcSalt in props.conf?

0 Karma


No, I'm not using a props.conf for this at all. How would it work with crcSalt?

0 Karma
Get Updates on the Splunk Community!

Index This | Why do they call it hyper text?

November 2023 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

State of Splunk Careers 2023: Career Resilience and the Continued Value of Splunk

For the past three years, Splunk has partnered with Enterprise Strategy Group to conduct a survey that gauges ...

The Great Resilience Quest: 9th Leaderboard Update

The ninth leaderboard update (11.9-11.22) for The Great Resilience Quest is out >> Kudos to all the ...