Getting Data In

How to monitor files with archiving policies

Zane
Explorer

I am currently encountering a problem where I have a log file that will be archived to another folder after reaching certain conditions. I have set up UF monitoring for both files, but the data collected may be duplicate. However, if I do not monitor the archive folder, some logs in the later positions will be lost in the file. I suspect it may be related to the file being archived too quickly? How to solve this problem

for example,my log file is abc.log, and then, it will be archived to current path /debug/abc.1.log, I have set the monitor for both files, but the data is duplicate, however,  if i do not monitor current path /debug/abc.1.log, i will lose the content at the end of the file.

 

 

Labels (4)
Tags (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Zane,

you could put under monitoring both the folders.

If you don't use the crcSal = <SOURCE> option, Splunk read only the last events in the rotated file and doesn't index twice the logs event if tey have a different filename (Remember that the above option must to be not present!).

Otherwise, if you rename the file before rotating (adding e.g. the new data to the file name), you can delay the rotation (30/60 seconds are sufficient) so Splunk will read also the last event in the file before i's moved to the new folder.

Ciao.

Giuseppe

Zane
Explorer

Hi  @gcusello 

thanks for your answer,

but I can‘t control delayed the rotation, due to those log file not managed by us, so if it's possible,adjusting from the Splunk side would be great.

so as you said, "you can delay the rotation (30/60 seconds are sufficient) so Splunk will read also the last event in the file",  according to this, i found there is a parameter in inputs.conf, "time_before_close", it's 3 by default,  can i adjust this value so as to delay UF close monitored files?for example, set it as 30?

thanks so much.

\Zane

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Zane,

no, this parameter is when you rotate a file on the same folder, but, if I correctly understood, yu move it in another folder.

The solution is to put under monitoring also the destination folder to tale only the events between the last read and the rotation, remembering that you cannot use crcSalt = <SOUCE> in your inputs.conf.

Ciao.

Giuseppe

Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

SignalFlow: What? Why? How?

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth ...

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...