Getting Data In

How does splunk eat newly copied edition of a file

crazyeva
Contributor

I mean e.g. if i manually copy and overwrite a "message.log" to splunk monitoring path, the new one contains some growth at end than the old one. How could i make sure splunk ignore the already indexed data, and just eat the increased part?

0 Karma
1 Solution

jeremiahc4
Builder

You need to ensure you are appending to the existing log rather than copying over it. When you copy a file over you are breaking Splunk's ability to keep track of it's pointer to where it left off.

If you are on Unix/Linux then you can use something like "cat mydata >> message.log" to append the contents of file "mydata" onto your message.log. Splunk should then be able to continue from it's pointer to the end of the file marker instead of re-reading the whole file.

View solution in original post

0 Karma

jeremiahc4
Builder

You need to ensure you are appending to the existing log rather than copying over it. When you copy a file over you are breaking Splunk's ability to keep track of it's pointer to where it left off.

If you are on Unix/Linux then you can use something like "cat mydata >> message.log" to append the contents of file "mydata" onto your message.log. Splunk should then be able to continue from it's pointer to the end of the file marker instead of re-reading the whole file.

0 Karma

crazyeva
Contributor

OK thank you

0 Karma

jeremiahc4
Builder

That sounds like the best way in your case.

If you are talking about syslog data on a Unix machine and you have root access, you could forward your syslog data. See "SUPPORT FOR REMOTE LOGGING" section at;
http://linux.about.com/od/commands/l/blcmdl8_syslogd.htm

I'm guessing if you can't install a forwarder, then you can't modify syslog.conf either though.

0 Karma

crazyeva
Contributor

Thank you, in my scene
splunk forwarder is not allowed on the log generating instance
I have to copy every new edition of the log file periodically
According to your suggestion, i do "cat message.log_new|tail -n (compared_lines) >> message.log"
Is there any better idea?

0 Karma
Get Updates on the Splunk Community!

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...