Control number of sources with rotated logfiles

[Starlette]

I am monitoring a dir with rotating logs, ( fi /depot/logs/ ) how can I control the source name, and avoid zillions of sources. (file_1.log file_2.log)

thanks! Starlette

[Lowell]

Just FYI, I've posted several fully functional source renaming transformers in another answer. (This is using the approach that southeringtonp is talking about.) Feel free to take a look and see if any of them will work for you: (Specifically, the transformer name "source_clean-digits-before-ext" looks like it will work for your situation.)

• Consolidate similarly named log files into a single source

[southeringtonp]

In inputs.conf, you can explicitly set the value of source for a given input definition:

[monitor:///var/log/something]
disabled = false
sourcetype = mysourcetype
source = mysource

Or, you can use a transform to assign it in a more targeted way:

[mysourcetype]
DEST_KEY = MetaData:Source
REGEX = (?=)
FORMAT = source::mysource

The above example will always set the source - adjust the REGEX setting as needed to match text in your events for a more targeted assignment.

[Starlette]

ah this looks promising,,,thanks!

[williamche]

You could try the following in your props.conf file to specify a sourcetype based on the file's naming convention:

[source::/depot/logs/file_*.log]
sourcetype = foo

[williamche]

Ah, I see what you did there! I must've read too much into Starlette's questions and thought that all the data from each log file were assigned to a unique sourcetype named after the filename. It happened to me when I left the sourcetype = automatic. So I used the method I suggested to overwrite the sourcetype so they are the same for all the rotated log files. (-2.. I have to make that up somehow! 🙂 )

[southeringtonp]

This sets sourcetype, not source.