Getting Data In

How does splunk eat newly copied edition of a file

crazyeva
Contributor

I mean e.g. if i manually copy and overwrite a "message.log" to splunk monitoring path, the new one contains some growth at end than the old one. How could i make sure splunk ignore the already indexed data, and just eat the increased part?

0 Karma
1 Solution

jeremiahc4
Builder

You need to ensure you are appending to the existing log rather than copying over it. When you copy a file over you are breaking Splunk's ability to keep track of it's pointer to where it left off.

If you are on Unix/Linux then you can use something like "cat mydata >> message.log" to append the contents of file "mydata" onto your message.log. Splunk should then be able to continue from it's pointer to the end of the file marker instead of re-reading the whole file.

View solution in original post

0 Karma

jeremiahc4
Builder

You need to ensure you are appending to the existing log rather than copying over it. When you copy a file over you are breaking Splunk's ability to keep track of it's pointer to where it left off.

If you are on Unix/Linux then you can use something like "cat mydata >> message.log" to append the contents of file "mydata" onto your message.log. Splunk should then be able to continue from it's pointer to the end of the file marker instead of re-reading the whole file.

0 Karma

crazyeva
Contributor

OK thank you

0 Karma

jeremiahc4
Builder

That sounds like the best way in your case.

If you are talking about syslog data on a Unix machine and you have root access, you could forward your syslog data. See "SUPPORT FOR REMOTE LOGGING" section at;
http://linux.about.com/od/commands/l/blcmdl8_syslogd.htm

I'm guessing if you can't install a forwarder, then you can't modify syslog.conf either though.

0 Karma

crazyeva
Contributor

Thank you, in my scene
splunk forwarder is not allowed on the log generating instance
I have to copy every new edition of the log file periodically
According to your suggestion, i do "cat message.log_new|tail -n (compared_lines) >> message.log"
Is there any better idea?

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.
Get Updates on the Splunk Community!

Introduction to Splunk AI

How are you using AI in Splunk? Whether you see AI as a threat or opportunity, AI is here to stay. Lucky for ...

Splunk + ThousandEyes: Correlate frontend, app, and network data to troubleshoot ...

Are you tired of troubleshooting delays caused by siloed frontend, application, and network data? We've got a ...

Maximizing the Value of Splunk ES 8.x

Splunk Enterprise Security (ES) continues to be a leader in the Gartner Magic Quadrant, reflecting its pivotal ...