Splunk Search

Control number of sources with rotated logfiles

Starlette
Contributor

I am monitoring a dir with rotating logs, ( fi /depot/logs/ ) how can I control the source name, and avoid zillions of sources. (file_1.log file_2.log)

thanks! Starlette

Lowell
Super Champion

Just FYI, I've posted several fully functional source renaming transformers in another answer. (This is using the approach that southeringtonp is talking about.) Feel free to take a look and see if any of them will work for you: (Specifically, the transformer name "source_clean-digits-before-ext" looks like it will work for your situation.)

0 Karma

southeringtonp
Motivator

In inputs.conf, you can explicitly set the value of source for a given input definition:

[monitor:///var/log/something]
disabled = false
sourcetype = mysourcetype
source = mysource

Or, you can use a transform to assign it in a more targeted way:

[mysourcetype]
DEST_KEY = MetaData:Source
REGEX = (?=)
FORMAT = source::mysource

The above example will always set the source - adjust the REGEX setting as needed to match text in your events for a more targeted assignment.

Starlette
Contributor

ah this looks promising,,,thanks!

0 Karma

williamche
Path Finder

You could try the following in your props.conf file to specify a sourcetype based on the file's naming convention:

[source::/depot/logs/file_*.log]
sourcetype = foo
0 Karma

williamche
Path Finder

Ah, I see what you did there! I must've read too much into Starlette's questions and thought that all the data from each log file were assigned to a unique sourcetype named after the filename. It happened to me when I left the sourcetype = automatic. So I used the method I suggested to overwrite the sourcetype so they are the same for all the rotated log files. (-2.. I have to make that up somehow! 🙂 )

0 Karma

southeringtonp
Motivator

This sets sourcetype, not source.

Get Updates on the Splunk Community!

Splunk Answers Content Calendar, July Edition I

Hello Community! Welcome to another month of Community Content Calendar series! For the month of July, we will ...

Secure Your Future: Mastering Upgrade Readiness for Splunk 10

Spotlight: The Splunk Health Assistant Add-On  The Splunk Health Assistant Add-On is your ultimate companion ...

Observability Unlocked: Kubernetes & Cloud Monitoring with Splunk IM

Ready to master Kubernetes and cloud monitoring like the pros? Join Splunk’s Growth Engineering team on ...