Getting Data In

Will monitored files that are renamed be reindexed?

Path Finder

I have an application that uses rolling logging. The rolling logging works as such:

  • logs are initially written to "log_file.txt"
  • after a certain time period, the "logfile.txt" is renamed to "logfile.txt.1", and an empty "log_file.txt" is created to start writing to.
  • after a certain time period, the "logfile.txt.1" is renamed to "logfile.txt.2", "logfile.txt" is renamed to "logfile.txt.1" and an empty "log_file.txt" is created to start writing to.
  • this process continues until X number of files are written, after that the oldest file file is deleted.

So if I monitor all of the files that match "log_file.*", will I have a bunch of duplicated data each time the file rolls over?

Should I just monitor "log_file.txt" in this instance?

0 Karma
1 Solution

Champion

The answer is somewhat "it depends", but the default settings are likely sane enough to prevent re-indexing old data. Splunk uses the first initCrcLength bytes of a file to determine if it is a new file or not. That setting can be changed as necessary, and other settings can also be changed to force Splunk to use the filename to assist it in determining uniqueness, but it sounds like you want the normal behavior that a monitor input would give you.

For more information on the different settings you can use for your inputs check out the inputs.conf spec.

View solution in original post

Champion

The answer is somewhat "it depends", but the default settings are likely sane enough to prevent re-indexing old data. Splunk uses the first initCrcLength bytes of a file to determine if it is a new file or not. That setting can be changed as necessary, and other settings can also be changed to force Splunk to use the filename to assist it in determining uniqueness, but it sounds like you want the normal behavior that a monitor input would give you.

For more information on the different settings you can use for your inputs check out the inputs.conf spec.

View solution in original post

SplunkTrust
SplunkTrust

Did you mean inputs.conf http://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf ?
props.conf only has initCrcLength but anything else is configurable in inputs.conf

cheers, MuS

0 Karma

Champion

Yes, thanks! I had linked to the correct page but put props for the text instead.

0 Karma

Esteemed Legend

It also very much depends on whether you are using crcSalt = <SOURCE> or not.

0 Karma