topic Re: Handle log rolling and index yesterday's log file in Getting Data In

Handle log rolling and index yesterday's log file

shabamichae — Tue, 04 Mar 2025 11:34:46 GMT

we have a scenario where we roll logs everyday. we want Splunk to index log file for yesterday only. We don't want to ingest todays log files. what specific setting d i require in my input. Conf file to only ingest yesterdays data.

ignoreOlderThan = 1d also ingests todays logfiles which i do not want to.

Re: Handle log rolling and index yesterday's log file

PickleRick — Tue, 04 Mar 2025 12:12:49 GMT

I think I'd try to simply use logrotate or some custom script to move the log from yesterday to another directory from which they would normally be ingested with monitor input.

Re: Handle log rolling and index yesterday's log file

livehybrid — Tue, 04 Mar 2025 14:00:17 GMT

Hi @shabamichae

What do your monitor stanzas currently look like for monitoring these files? Do the logs roll to a "logName.log.1" format (.1 being yesterday)?

If so. you may be able to update your existing monitor stanzas to add a whitelist (see https://docs.splunk.com/Documentation/Splunk/9.4.0/Data/Monitorfilesanddirectorieswithinputs.conf#:~:text=whitelist%20%3D%20%3Cregular%20expression%3E)

whitelist = <regular expression> If set, the Splunk platform monitors files whose names match the specified regular expression. ## inputs.conf ## [monitor:///var/log/*] index=syslog sourcetype=example ..etc.. whitelist = .*\.1$

Also check out https://docs.splunk.com/Documentation/Splunk/latest/Data/Specifyinputpathswithwildcards

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

Re: Handle log rolling and index yesterday's log file

shabamichae — Wed, 05 Mar 2025 12:21:43 GMT

Hi @livehybrid
Thanks for your response, below is a sample log file names

server.log.20250303.1
server.log.20250303.10
server.log.20250303.11
server.log.20250303.12
server.log.20250303.13
server.log.20250303.14
server.log.20250303.15

Re: Handle log rolling and index yesterday's log file

shabamichae — Wed, 05 Mar 2025 12:26:11 GMT

Yes i do understand would require some kind of regex , but My issue is how do i wrrite the regex to match the date , do i need to configure a dat.xml file to read the current date

server.log.20250303.1
server.log.20250303.10
server.log.20250303.11
server.log.20250303.12
server.log.20250303.13
server.log.20250303.14
server.log.20250303.15

Re: Handle log rolling and index yesterday's log file

PickleRick — Wed, 05 Mar 2025 19:52:09 GMT

You simply can't. A regex matches a pattern. The pattern is static. It can contain some "recursive" elements but you can't put something like "today's date" as part of the pattern.