topic Re: is there a way to tail a file to index any new changes? in Getting Data In

is there a way to tail a file to index any new changes?

tantingli — Sun, 04 Apr 2010 23:08:35 GMT

I let splunk monitor a directory of files. I found when any file got changed splunk will reindex all events in the file, not just the new lines added. Is there a way to "tail" those files so splunk will reindex/import only the new lines instead of reloading all event in the files?

Re: is there a way to tail a file to index any new changes?

the_wolverine — Mon, 05 Apr 2010 00:11:30 GMT

Yes, the followTail setting tells Splunk to index only new events in a monitored file:

followTail = 0|1

If set to 1, monitoring begins at the end of the file (like tail -f).

This only applies to files the first time they are picked up. * After that, Splunk's internal file position records keep track of the file.

http://www.splunk.com/base/Documentation/4.0.10/Admin/Monitorfilesanddirectories#Monitor_syntax_and_examples

Re: is there a way to tail a file to index any new changes?

tantingli — Mon, 05 Apr 2010 00:36:53 GMT

Thanks! I tried it and it works for tailing the file.
But I found two problems:
1. say if I have 1.log in the directory. adding new lines works fine. but if I overwrite the existing content with new events then it won't get import/indexed.
2. if I copy another file to this directory called 2.log. That file won't get picked up automatically.

These 2 problems were not there without followTail=1.

Re: is there a way to tail a file to index any new changes?

tantingli — Mon, 05 Apr 2010 02:26:29 GMT

I found a solution:

[monitor:///LOGS/dtjlogs]
disabled = false
host =
host_regex =
host_segment = 3
index = default
sourcetype =

[monitor:///LOGS/dtjlogs/*/*.log]
disabled = false
host =
host_regex =
host_segment = 3
index = default
sourcetype =
followTail = 1

actually the two [monitor] are monitoring same files The only difference is the followTail=1 I found if I name the two monitor the same, only the first stanza works So just work-around by naming them slightly different. Now any new files will be picked up by the followTail=0 stanza and appended(tailing) events will be picked up by followTail=1 stanza. Ha, what a solution.

Re: is there a way to tail a file to index any new changes?

jrodman — Tue, 06 Apr 2010 03:45:08 GMT

Don't rely too much on this. The behavior for overlapping inputs like this is not well explored in 4.0. I suspect just the followtail stanza is needed, but you may not have been testing the file modification in a realistic way.

Be sure to replace the contents with new contents, rather than modifying a few lines in the middle. Also in 4.0, be prepared to be somewhat patient. In 4.1, the requirement for patience is much reduced.

Re: is there a way to tail a file to index any new changes?

jrodman — Tue, 06 Apr 2010 03:45:12 GMT

In 4.1, overlapping stanzas are supported, in this case, the .log files in subdirectories are handled by the second stanza, while the other files are handled by the first.

Re: is there a way to tail a file to index any new changes?

the_wolverine — Thu, 08 Apr 2010 06:16:06 GMT

From my testing, it looks like followTail is working. Files in subdirectories of the configured monitor input directory are properly recognized when If it is not working for you, please submit a ticket to support and provide your test case.

Re: is there a way to tail a file to index any new changes?

jrodman — Sat, 17 Apr 2010 08:47:44 GMT

Agree with Tina, problem 1 is not known, and probably does not occur. When manually testing monitor behavior, it's easy to not trigger reindexing, because splunk looks for the behavior of real logging programs (the whole file is changed, not just portions).

Problem 2 is known. Followtail's implementation assumes that you roll existing files rather than add new ones. You could use followtail on specific individual file paths to avoid this problem.

Re: is there a way to tail a file to index any new changes?

jrodman — Sat, 17 Apr 2010 08:49:09 GMT

Can you please consider modifying this question to clarify either what you were trying to do, or to inquire about the strange behavior you saw? The goal is for these to be informative to others.