Splunk Search

Control number of sources with rotated logfiles

Starlette
Contributor

I am monitoring a dir with rotating logs, ( fi /depot/logs/ ) how can I control the source name, and avoid zillions of sources. (file_1.log file_2.log)

thanks! Starlette

Lowell
Super Champion

Just FYI, I've posted several fully functional source renaming transformers in another answer. (This is using the approach that southeringtonp is talking about.) Feel free to take a look and see if any of them will work for you: (Specifically, the transformer name "source_clean-digits-before-ext" looks like it will work for your situation.)

0 Karma

southeringtonp
Motivator

In inputs.conf, you can explicitly set the value of source for a given input definition:

[monitor:///var/log/something]
disabled = false
sourcetype = mysourcetype
source = mysource

Or, you can use a transform to assign it in a more targeted way:

[mysourcetype]
DEST_KEY = MetaData:Source
REGEX = (?=)
FORMAT = source::mysource

The above example will always set the source - adjust the REGEX setting as needed to match text in your events for a more targeted assignment.

Starlette
Contributor

ah this looks promising,,,thanks!

0 Karma

williamche
Path Finder

You could try the following in your props.conf file to specify a sourcetype based on the file's naming convention:

[source::/depot/logs/file_*.log]
sourcetype = foo
0 Karma

williamche
Path Finder

Ah, I see what you did there! I must've read too much into Starlette's questions and thought that all the data from each log file were assigned to a unique sourcetype named after the filename. It happened to me when I left the sourcetype = automatic. So I used the method I suggested to overwrite the sourcetype so they are the same for all the rotated log files. (-2.. I have to make that up somehow! 🙂 )

0 Karma

southeringtonp
Motivator

This sets sourcetype, not source.

Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In January, the Splunk Threat Research Team had one release of new security content via the Splunk ES Content ...

Expert Tips from Splunk Professional Services, Ensuring Compliance, and More New ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Observability Release Update: AI Assistant, AppD + Observability Cloud Integrations & ...

This month’s releases across the Splunk Observability portfolio deliver earlier detection and faster ...