Getting Data In

How does splunk eat newly copied edition of a file

crazyeva
Contributor

I mean e.g. if i manually copy and overwrite a "message.log" to splunk monitoring path, the new one contains some growth at end than the old one. How could i make sure splunk ignore the already indexed data, and just eat the increased part?

0 Karma
1 Solution

jeremiahc4
Builder

You need to ensure you are appending to the existing log rather than copying over it. When you copy a file over you are breaking Splunk's ability to keep track of it's pointer to where it left off.

If you are on Unix/Linux then you can use something like "cat mydata >> message.log" to append the contents of file "mydata" onto your message.log. Splunk should then be able to continue from it's pointer to the end of the file marker instead of re-reading the whole file.

View solution in original post

0 Karma

jeremiahc4
Builder

You need to ensure you are appending to the existing log rather than copying over it. When you copy a file over you are breaking Splunk's ability to keep track of it's pointer to where it left off.

If you are on Unix/Linux then you can use something like "cat mydata >> message.log" to append the contents of file "mydata" onto your message.log. Splunk should then be able to continue from it's pointer to the end of the file marker instead of re-reading the whole file.

0 Karma

crazyeva
Contributor

OK thank you

0 Karma

jeremiahc4
Builder

That sounds like the best way in your case.

If you are talking about syslog data on a Unix machine and you have root access, you could forward your syslog data. See "SUPPORT FOR REMOTE LOGGING" section at;
http://linux.about.com/od/commands/l/blcmdl8_syslogd.htm

I'm guessing if you can't install a forwarder, then you can't modify syslog.conf either though.

0 Karma

crazyeva
Contributor

Thank you, in my scene
splunk forwarder is not allowed on the log generating instance
I have to copy every new edition of the log file periodically
According to your suggestion, i do "cat message.log_new|tail -n (compared_lines) >> message.log"
Is there any better idea?

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud’s AI Assistant in Action Series: Analyzing and ...

This is the second post in our Splunk Observability Cloud’s AI Assistant in Action series, in which we look at ...

Elevate Your Organization with Splunk’s Next Platform Evolution

 Thursday, July 10, 2025  |  11AM PDT / 2PM EDT Whether you're managing complex deployments or looking to ...

Splunk Answers Content Calendar, June Edition

Get ready for this week’s post dedicated to Splunk Dashboards! We're celebrating the power of community by ...