Getting Data In

How does splunk eat newly copied edition of a file

crazyeva
Contributor

I mean e.g. if i manually copy and overwrite a "message.log" to splunk monitoring path, the new one contains some growth at end than the old one. How could i make sure splunk ignore the already indexed data, and just eat the increased part?

0 Karma
1 Solution

jeremiahc4
Builder

You need to ensure you are appending to the existing log rather than copying over it. When you copy a file over you are breaking Splunk's ability to keep track of it's pointer to where it left off.

If you are on Unix/Linux then you can use something like "cat mydata >> message.log" to append the contents of file "mydata" onto your message.log. Splunk should then be able to continue from it's pointer to the end of the file marker instead of re-reading the whole file.

View solution in original post

0 Karma

jeremiahc4
Builder

You need to ensure you are appending to the existing log rather than copying over it. When you copy a file over you are breaking Splunk's ability to keep track of it's pointer to where it left off.

If you are on Unix/Linux then you can use something like "cat mydata >> message.log" to append the contents of file "mydata" onto your message.log. Splunk should then be able to continue from it's pointer to the end of the file marker instead of re-reading the whole file.

0 Karma

crazyeva
Contributor

OK thank you

0 Karma

jeremiahc4
Builder

That sounds like the best way in your case.

If you are talking about syslog data on a Unix machine and you have root access, you could forward your syslog data. See "SUPPORT FOR REMOTE LOGGING" section at;
http://linux.about.com/od/commands/l/blcmdl8_syslogd.htm

I'm guessing if you can't install a forwarder, then you can't modify syslog.conf either though.

0 Karma

crazyeva
Contributor

Thank you, in my scene
splunk forwarder is not allowed on the log generating instance
I have to copy every new edition of the log file periodically
According to your suggestion, i do "cat message.log_new|tail -n (compared_lines) >> message.log"
Is there any better idea?

0 Karma
Get Updates on the Splunk Community!

Dashboards: Hiding charts while search is being executed and other uses for tokens

There are a couple of features of SimpleXML / Classic dashboards that can be used to enhance the user ...

Splunk Observability Cloud's AI Assistant in Action Series: Explaining Metrics and ...

This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how ...

Brains, Bytes, and Boston: Learn from the Best at .conf25

When you think of Boston, you might picture colonial charm, world-class universities, or even the crack of a ...