Getting Data In

Forwarder fails, monitored file is archived, what happens?

Path Finder

Hello,

I have a hypothetical scenario which I hope someone can help me with.

Let's say I have a Linux server with a universal forwarder installed which is monitoring /var/log/messages.

Imagine the universal forwarder now fails for some reason - someone accidentally killed the process, etc.

Now when the universal forwarder has failed, /var/log/messages has been written to its maximum size and rolled-over to /var/log/messages-20120727, and a new /var/log/messages has been created.

We now discover that the universal forwarder has stopped, and we restart it, and it happily monitors /var/log/messages again.

Would I be right to say that whatever changes was written to the previous /var/log/messages file after the universal forwarder failed, is now lost, i.e. not sent to the indexer?

The Splunk forwarder maintains a pointer to what data it has forwarded to an indexer in the fishbucket index for /var/log/messages, but what happens now that /var/log/messages is essentially an entirely new file?

Thanks in advance for any help!

1 Solution

Legend

I think you'll find this docs page very useful, as it explains some things about how Splunk handles rotated log files.

http://docs.splunk.com/Documentation/Splunk/latest/Data/Howlogfilerotationishandled

Essentially, if you've told Splunk to monitor just /var/log/messages, then obviously it won't see events in another file. If, however, you choose to monitor the whole /var/log directory, or for that matter just all /var/log/messages* files, Splunk will identify file contents based on the actual contents, not the filename, so if you move /var/log/messages to /var/log/messages-20120727 it'll just pick up where it left off.

As it says in the docs page though, doing this with .gz compressed files is another thing, but you didn't specify that in your scenario so I'll omit that discussion 🙂

View solution in original post

Legend

I think you'll find this docs page very useful, as it explains some things about how Splunk handles rotated log files.

http://docs.splunk.com/Documentation/Splunk/latest/Data/Howlogfilerotationishandled

Essentially, if you've told Splunk to monitor just /var/log/messages, then obviously it won't see events in another file. If, however, you choose to monitor the whole /var/log directory, or for that matter just all /var/log/messages* files, Splunk will identify file contents based on the actual contents, not the filename, so if you move /var/log/messages to /var/log/messages-20120727 it'll just pick up where it left off.

As it says in the docs page though, doing this with .gz compressed files is another thing, but you didn't specify that in your scenario so I'll omit that discussion 🙂

View solution in original post

New Member

Could you explain what happens in the case where files are compressed when they are archived?

0 Karma

Contributor

May be an old issue ...this question was about file.
how to handle data loss in case of TCP input ?

I want to handle universal forwarder failure in a multisite deployment scenario !!!

0 Karma

Path Finder

Thanks for the help 🙂 Guess I have to monitor all the files just in case.

0 Karma

Contributor

May be an old issue ...this question was about file.
how to handle data loss in case of TCP input ?

I want to handle universal forwarder failure in a multisite deployment scenario !!!

0 Karma

Influencer

[monitor:///var/log/messages*] should do that.

0 Karma

Legend

Like I said, yes, if you only monitor /var/log/messages then that's what will happen. The events that were written after the forwarder went down (though I must add that I've hardly ever seen forwarders do that) will no longer be in /var/log/messages when the forwarder comes back up, so it has no way of seeing the events that it "missed". That is one of the reasons why you should make sure that if you are rotating your logfiles, you should monitor not just the non-rotated file but also the rotated ones.

0 Karma

Path Finder

Hello Ayn,

Thanks for the reply, please let me clarify.

Let's assume we are only monitoring one file, /var/log/messages, with the following sequence of events.

  1. Forwarder fails.
  2. /var/log/messages is written to, and rotated to /var/log/messages1
  3. A new /var/log/messages is created.
  4. Forwarder is restarted.
  5. Forwarder now treats /var/log/messages as a new file because begin and end CRC have not been previously seen.

So, in this case, some events would not be indexed, because the forwarder was not active before /var/log/messages was rotated, right?

0 Karma