Monitor advice needed

[kenison]

After reading the docs and looking in forums, I thought I had a understanding of monitor and what it does...I guess not.
Is a monitor, set up to follow from tail, only supposed to index data that is written to a directory from the time of monitor creation? I manually made a monitor in my inputs.conf file after I saw that the monitor I set up in the manager was grabbing events that were pre-dated.

[monitor:///dir/path]
blacklist = dir/
followTail = 1

I didn't see events logging right away after restarting splunk, so I thought it was working properly...that is, only indexing events that are new. I came in today to find my license had exceeded limit over night and splunk has indexed events from last year.

Someone tell me what is wrong with this. Is there a way to set up a monitor that only indexes new events? Why is my monitor indexing the whole file?

[MuS]

Hi kenison.vrabcak

well there is nothing wrong, this is the way splunk monitors directories: starting at the moment you add it in a monitor [stanza] splunk is 'eating' up any readable file in this directory. How should splunk know what you consider as old data?

if you really want to have only new data to be indexed, move the 'old logs' out of the way before you [monitor] the directory.

after that splunk will index only the new files coming in and will forget about the already indexed files.

regards

[mzorzi]

With the option followTail enabled Splunk is going to monitor only events being added into the monitored stanza after restarting Splunk. Maybe the old files are in compress format and their modification time has been changed. Or maybe you have an incorrect timestamp extraction problem, and the events are not really from last year.

You can find more information on how to troubleshoot this problem by reviewing the content of this twiki page:

http://www.splunk.com/wiki/Community:Troubleshooting\_Monitor_Inputs

and this answer:

http://splunk-base.splunk.com/answers/1162/is-there-some-way-to-see-the-current-tailing-status-of-my...