topic Re: monitor config for log files - universal forwarder in Getting Data In

monitor config for log files - universal forwarder

lakshman237 — Thu, 12 Jul 2012 15:26:29 GMT

I have log files, say, "logFile1.txt", "logFile2.txt" in folder /home/system/logs/ . The folder also has rotated logs which are of the form "logFile1.201206021010.txt" ( yyyymmddhhmm) added.
[monitor:////home/system/logs/logFile*.txt]
disabled = false
sourcetype = mysystem
index = myindex

The above config brings rotated logs well to the index and sourcetype, which I donot want. I can add two stanza's one for logFile1.txt and another for logFile2.txt. However is there a better way to do this?

Re: monitor config for log files - universal forwarder

sdaniels — Thu, 12 Jul 2012 15:39:53 GMT

You probably want to add an entry to inputs.conf for crcSalt.

crcSalt=<\SOURCE>

crcSalt = /
* Use this setting to force Splunk to consume files that have matching CRCs (cyclic redundancy checks). (Splunk only
performs CRC checks against the first few lines of a file. This behavior prevents Splunk from indexing the same
file twice, even though you may have renamed it -- as, for example, with rolling log files. However, because the
CRC is based on only the first few lines of the file, it is possible for legitimately different files to have
matching CRCs, particularly if they have identical headers.)
* If set, is added to the CRC.
* If set to the literal string (including the angle brackets), the full directory path to the source file
is added to the CRC. This ensures that each file being monitored has a unique CRC. When crcSalt is invoked,
it is usually set to .
* Be cautious about using this attribute with rolling log files; it could lead to the log file being re-indexed
after it has rolled.
* Defaults to empty.

Re: monitor config for log files - universal forwarder

lakshman237 — Thu, 12 Jul 2012 16:43:00 GMT

thanks, but will this ensure logFile1.txt and LogFile2.txt are indexed but not the rotated files? ( with the above monitor command). I had an issue with double indexing in the past with crcSalt. let me check this again.

Re: monitor config for log files - universal forwarder

sdaniels — Thu, 12 Jul 2012 20:08:35 GMT

Let me know what you find out. Logs with very large headers cause problems since Splunk doesn't detect any change. This is addressed in our next major version.

Re: monitor config for log files - universal forwarder

Ayn — Fri, 13 Jul 2012 05:44:23 GMT

This docs section covers how Splunk handles rotated files. Essentially, when you initially add a directory for monitoring Splunk will read all of the files in there because it hasn't seen any of them before, but after that it will never re-index a rotated file because the contents will be the same as before.

http://docs.splunk.com/Documentation/Splunk/latest/Data/Howlogfilerotationishandled