Monitoring Splunk

Monitor stanza in inputs.conf file for rotating log files with access_log.todaysdate as the name.

saad
Loves-to-Learn

I want to monitor a log file which gets created everyday with new day date in its name.

If I configure inputs.conf as below it will monitor all the older log files of previous days as well which will flood my source list, i want only the latest todays logfile to be monitored. Please suggest.

 

[monitor:////path to direct/access_log.*]
sourcetype = log4j
ignoreOlderThan = 7d
crcSalt = <string>

Labels (1)
0 Karma

saad
Loves-to-Learn

I guess you dint get my querry. I have a log file which get created daily with todays date. for eg todays logfile name is access_log.2021-09-22. A new file will be created tomm "access_log.2021-09-23". I want to monitor only todays file and not all the previous dates file in that directory.

0 Karma

codebuilder
Influencer

Are these OS logs, or application specific?

If specific to your application and you're unable to change how new logs are named (eliminate the timestamp), then you would have to find a workaround. One option would be to configure logrotate so that the OS rotates and compresses your logs daily, then you could blacklist .gz files.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

PickleRick
SplunkTrust
SplunkTrust

From the configuration I suspect it's some java application. Might be a tomcat or jboss app. They are ugly by design and very troublesome to do log rotation properly.

0 Karma

codebuilder
Influencer

Based on the info in your post, Linux is already rotating your logs as it should. So all you really need to do is remove the wildcard and instead specify the file name with extension (access_log.log e.g.).

Splunk will then monitor that file in real time and pick back up on it after the OS rotates it while ignoring any other files in that directory. access_log.log-20210922, access_log.log.1 or access_log.gz would all be ignored, for example.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

saad
Loves-to-Learn

Thank you for your response.

But unfortunately this would not work as i dont have any file with access_log.log (my log file get created daily with that days date)

I have a log file which gets created daily with todays date. for eg todays logfile name is "access_log.2021-09-22". A new file will be created tomm "access_log.2021-09-23". I want to monitor only todays file and not all the previous dates file in that directory.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Firstly, 7d is quite low. You should set a value high enough so that your files should never reach, including maintenances, downtimes and so on. But then again maybe in your case 7d is fine.

Anyway, ignoreOlderThan works by checking file attributes (mtime if I remember correctly) so make sure that your files do have this attribute set to the past. If you are unable to meet this condition (because you're exporting the files over the network and the NFS/CIFS/whatever is cheating on the times or you have a file rotation mechanism that does update the attributes every night) I'm afraid your only hope is to craft a regex to blacklist some of the files.

Oh, and btw, it seems like wrong section of the forum. 😉

0 Karma
Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...