topic Re: Control number of sources with rotated logfiles in Splunk Search

Control number of sources with rotated logfiles

Starlette — Fri, 15 Oct 2010 18:02:11 GMT

I am monitoring a dir with rotating logs, ( fi /depot/logs/ ) how can I control the source name, and avoid zillions of sources. (file_1.log file_2.log)

thanks! Starlette

Re: Control number of sources with rotated logfiles

williamche — Fri, 15 Oct 2010 20:11:47 GMT

You could try the following in your props.conf file to specify a sourcetype based on the file's naming convention:

[source::/depot/logs/file_*.log]
sourcetype = foo

Re: Control number of sources with rotated logfiles

southeringtonp — Fri, 15 Oct 2010 21:05:26 GMT

In inputs.conf, you can explicitly set the value of source for a given input definition:

[monitor:///var/log/something]
disabled = false
sourcetype = mysourcetype
source = mysource

Or, you can use a transform to assign it in a more targeted way:

[mysourcetype]
DEST_KEY = MetaData:Source
REGEX = (?=)
FORMAT = source::mysource

The above example will always set the source - adjust the REGEX setting as needed to match text in your events for a more targeted assignment.

Re: Control number of sources with rotated logfiles

southeringtonp — Fri, 15 Oct 2010 21:06:22 GMT

This sets sourcetype, not source.

Re: Control number of sources with rotated logfiles

Starlette — Fri, 15 Oct 2010 23:51:04 GMT

ah this looks promising,,,thanks!

Re: Control number of sources with rotated logfiles

Lowell — Sat, 16 Oct 2010 03:11:51 GMT

Just FYI, I've posted several fully functional source renaming transformers in another answer. (This is using the approach that southeringtonp is talking about.) Feel free to take a look and see if any of them will work for you: (Specifically, the transformer name "source_clean-digits-before-ext" looks like it will work for your situation.)

Consolidate similarly named log files into a single source

Re: Control number of sources with rotated logfiles

williamche — Tue, 19 Oct 2010 01:04:05 GMT

Ah, I see what you did there! I must've read too much into Starlette's questions and thought that all the data from each log file were assigned to a unique sourcetype named after the filename. It happened to me when I left the sourcetype = automatic. So I used the method I suggested to overwrite the sourcetype so they are the same for all the rotated log files. (-2.. I have to make that up somehow! 🙂 )