topic Re: How do I ingest rotated log files without the source filename changing? in Getting Data In

How do I ingest rotated log files without the source filename changing?

marrette — Tue, 29 Sep 2020 21:34:10 GMT

I have several logs files on several hosts which ingest data from log files which are quite high volume (nearly as high as 2gb/hour on a big day) and are rotated every hour. On rotation the file name will change from Device_01.log to Device_01.log.2018-10-08-12

The inputs.conf stanza is configured like so:

[monitor:///path/to/logs/...]
disabled = false
followTail = 0
index = myIndex
whitelist =  .*\.log$|.*\.log\.\d\d\d\d-\d\d-\d\d-\d\d$
ignoreOlderThan = 1d
blacklist = .*ffdc_.*log|messages_.*log|exception_.*log|trace.*log|native*.log|activity.log|systemout*[0-9].log|systemout_.*log|.*-metrics\.[0-9]{1,3}\.log|\d+.\d+.\d+.\d+.\d+.\d+.log|-\d+-\d+-\d+.log
sourcetype = myApplication

And this works - if Splunk isn't able to keep up with the data coming in before the file is rotated on the hour, it will open the renamed file and read the rest of the data from where it left off. But the source field of the renaming data will be the renamed file, not the actually log file name. So the following Splunk query will show the whole file if a wildcard is used:

host=AppServer* index=myIndex source=/path/to/logs/Device_01.log*

...but it would be nice if Splunk would keep the source field named with the original file name, not the rolled filename.

Is this possible to do?

Thanks
Eddie

Re: How do I ingest rotated log files without the source filename changing?

FrankVl — Mon, 08 Oct 2018 06:44:43 GMT

You could create a props and transforms config to overwrite the value of the source field. Try something like below (regex might require some tweaking). Deploy this on your indexer(s) (or on your heavy forwarder if you use one for this data).

props.conf

[myApplication]
TRANSFORMS-setsource = myApp-setsource

transforms.conf

[myApp-setsource]
SOURCE_KEY = MetaData:Source
REGEX = (^[^\.]+\.log).*
DEST_KEY = MetaData:Source
FORMAT = source::$1

https://regex101.com/r/lcKD4Y/1

Re: How do I ingest rotated log files without the source filename changing?

mstjohn_splunk — Mon, 08 Oct 2018 19:09:21 GMT

hi @marrette,

Did the answer below solve your problem? If so, please resolve this post by approving it! If your problem is still not solved, keep us updated so that someone else can help ya.

Thanks for posting!