Getting Data In

How to edit my universal forwarder's monitor configuration for a single log file to prevent indexing events over and over again?

lukas_loder
Communicator

Hello,

We try to monitor a single Logfile with a Splunk Universal Forwarder on a Windows Server 2008 R2 Server. In this Logfile, the newest Events always get posted at the top of the file.

If I use a Basic Setting like this:

[monitor://D:\...\folder\]
index = app
sourcetype = System
recursive = false
whitelist = Filename.log
blacklist = otherFilename
disabled=0

It works fine first, but then it starts logging all Events over and over again. In the Splunkd.log i get following error:

03-24-2015 10:31:22.040 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='D:\...forder\Filename.log'.

If I try the Option followTail=1 or followTail=true, it doesn't work anymore. It doesn't send anything to my Splunk indexer.

Does someone know this problem or is there a default solution? Unfortunately, I couldn't find a parameter to change the order of the logfile.

Thanks!

0 Karma

lguinn2
Legend

This is going to be a problem for Splunk, which expects the newest events to be at the end of the file.

Whenever Splunk sees that the beginning of a file has changed, it assumes that it is a new file and re-indexes the whole thing. This is what is happening to this file now. Using crcSalt would turn off this behavior - BUT it will not make Splunk index the new events only.

I don't know of any Splunk settings which would properly configure an input like this. My only suggestion is this: write a script that periodically reviews the log and extracts only the new events and sends them to Splunk. Hopefully someone else has a better idea.

Or, fix the logging so that it writes to the end of the file.

satishsdange
Builder

Are you using crcSalt in props.conf?

0 Karma

lukas_loder
Communicator

No, I'm not using a props.conf for this at all. How would it work with crcSalt?

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...