Getting Data In

How do I ingest rotated log files without the source filename changing?

marrette
Path Finder

I have several logs files on several hosts which ingest data from log files which are quite high volume (nearly as high as 2gb/hour on a big day) and are rotated every hour. On rotation the file name will change from Device_01.log to Device_01.log.2018-10-08-12

The inputs.conf stanza is configured like so:

[monitor:///path/to/logs/...]
disabled = false
followTail = 0
index = myIndex
whitelist =  .*\.log$|.*\.log\.\d\d\d\d-\d\d-\d\d-\d\d$
ignoreOlderThan = 1d
blacklist = .*ffdc_.*log|messages_.*log|exception_.*log|trace.*log|native*.log|activity.log|systemout*[0-9].log|systemout_.*log|.*-metrics\.[0-9]{1,3}\.log|\d+.\d+.\d+.\d+.\d+.\d+.log|-\d+-\d+-\d+.log
sourcetype = myApplication

And this works - if Splunk isn't able to keep up with the data coming in before the file is rotated on the hour, it will open the renamed file and read the rest of the data from where it left off. But the source field of the renaming data will be the renamed file, not the actually log file name. So the following Splunk query will show the whole file if a wildcard is used:

host=AppServer* index=myIndex source=/path/to/logs/Device_01.log*

...but it would be nice if Splunk would keep the source field named with the original file name, not the rolled filename.

Is this possible to do?

Thanks
Eddie

0 Karma
1 Solution

FrankVl
Ultra Champion

You could create a props and transforms config to overwrite the value of the source field. Try something like below (regex might require some tweaking). Deploy this on your indexer(s) (or on your heavy forwarder if you use one for this data).

props.conf

[myApplication]
TRANSFORMS-setsource = myApp-setsource

transforms.conf

[myApp-setsource]
SOURCE_KEY = MetaData:Source
REGEX = (^[^\.]+\.log).*
DEST_KEY = MetaData:Source
FORMAT = source::$1

https://regex101.com/r/lcKD4Y/1

View solution in original post

mstjohn_splunk
Splunk Employee
Splunk Employee

hi @marrette,

Did the answer below solve your problem? If so, please resolve this post by approving it! If your problem is still not solved, keep us updated so that someone else can help ya.

Thanks for posting!

0 Karma

FrankVl
Ultra Champion

You could create a props and transforms config to overwrite the value of the source field. Try something like below (regex might require some tweaking). Deploy this on your indexer(s) (or on your heavy forwarder if you use one for this data).

props.conf

[myApplication]
TRANSFORMS-setsource = myApp-setsource

transforms.conf

[myApp-setsource]
SOURCE_KEY = MetaData:Source
REGEX = (^[^\.]+\.log).*
DEST_KEY = MetaData:Source
FORMAT = source::$1

https://regex101.com/r/lcKD4Y/1

Get Updates on the Splunk Community!

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...