How does splunk eat newly copied edition of a file

[crazyeva]

I mean e.g. if i manually copy and overwrite a "message.log" to splunk monitoring path, the new one contains some growth at end than the old one. How could i make sure splunk ignore the already indexed data, and just eat the increased part?

[jeremiahc4 · Accepted solution]

You need to ensure you are appending to the existing log rather than copying over it. When you copy a file over you are breaking Splunk's ability to keep track of it's pointer to where it left off.

If you are on Unix/Linux then you can use something like "cat mydata >> message.log" to append the contents of file "mydata" onto your message.log. Splunk should then be able to continue from it's pointer to the end of the file marker instead of re-reading the whole file.

[crazyeva]

OK thank you

[jeremiahc4]

That sounds like the best way in your case.

If you are talking about syslog data on a Unix machine and you have root access, you could forward your syslog data. See "SUPPORT FOR REMOTE LOGGING" section at;
http://linux.about.com/od/commands/l/blcmdl8_syslogd.htm

I'm guessing if you can't install a forwarder, then you can't modify syslog.conf either though.

[crazyeva]

Thank you, in my scene
splunk forwarder is not allowed on the log generating instance
I have to copy every new edition of the log file periodically
According to your suggestion, i do "cat message.log_new|tail -n (compared_lines) >> message.log"
Is there any better idea?