topic Re: monitor files query and configuration in Monitoring Splunk

monitor files query and configuration

shaunwilde — Mon, 28 Sep 2020 11:28:36 GMT

I am trying to configure Splunk to monitor a service that has some unusual (to me) logging behaviour (unfortunately re-writing the logging code is not an option as it is 3rd party)

In folder X it creates several files

A.log
B.log
C.log

and appends to them.

After 24 hours it

moves/copies A.log to old/A__< date_stamp >.log
moves/copies B.log to old/B__< date_stamp >.log
moves/copies C.log to old/C__< date_stamp >.log

and truncates/creates the original file again

Questions:

If I just monitor the original files i.e. A.log, ..., would it be possible that splunk may miss a logging event due to the event being written and the file being copied/truncated?

If I try to monitor all the log files (so I never loose a message) can I wildcard it such that all A*.log events have a common sourcetype?

If I can end up watching all the log files it is possible I'll have duplicate entries 1st from the orginal and the 2nd from the copy into the old folder and if so is there a common/known strategy to handle that within splunk?

Re: monitor files query and configuration

bwooden — Mon, 05 Mar 2012 04:44:30 GMT

You probably want to monitor via a wildcard, unless your process pauses before rotating logs.

Why? If you use a wildcard to monitor a path, Splunk will index duplicates. Splunk calculates a CRC of the file and uses that when keeping track of how far into a file it has read. If the log is rotated, but still included in them monitor stanza (i.e. via a wildcard) Splunk will know it has already read most of that file and only read anything it hasn't already indexed.

NB: Caveats - This is not true if files are compressed as part of rotation. If a wildcard includes compressed files, they should be blacklisted. A generic blacklist might look something like this:

blacklist = \.(tar|gz|bz2|tar.gz|tgz|tbz|tbz2|zip|z)$

Using crcSalt= would also break this model. For a more complete explanation of how Splunk handles log file rotation, check out said topic in the Getting Things in Manual.

Re: monitor files query and configuration

shaunwilde — Mon, 05 Mar 2012 05:07:26 GMT

So you are saying I should do something like this

[monitor:////data/X/logs/.../A*.log]
index = my_index
sourcetype = product.a.log

and that this will track A because the CRC of the header in old A.log and in A+date.log will be the same and so will notice extra entries appended t the bottom if they don't match? Is there a way to change the size of the CRC header size? The is a lot of xml being logged and it could be that the last 256 do not change between messages (though the file size will)