All Apps and Add-ons

Splunk not ingest most recent logs.

arun_kant_sharm
Path Finder

Hi Experts,

I try to ingest data from a mount point, in that path all files are log rotate and moved to .gz format on daily basis.
But my splunk ingest only that file that already converted into .gz file. So before 24 hours I don't find latest logs.
Please suggest how i get latest logs in the Splunk.

Sample conf files are :
inputs.conf :
[monitor:///data/production/logs/JobBatch/documents]
disabled = false
index = logs_jobbatch
sourcetype = JobBatch
whitelist = (zipDocument_[\d]+.log.gz|zipDocument_[\d]+.log)

props.conf :
[JobBatch]
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = false
category = Custom
disabled = false
pulldown_type = true
BREAK_ONLY_BEFORE = ^\d{4}-\d{2}-\d{2}\s{1}\d{2}:\d{2}:\d{2}.\d{3}\s{1}
HEADER_FIELD_LINE_NUMBER = 9
TRUNCATE = 750000
MAX_EVENTS = 1000
BREAK_ONLY_BEFORE_DATE =
SHOULD_LINEMERGE = false

0 Karma

DavidHourani
Super Champion

Hi @arun_kant_sharma,

You're ingesting both rotated files and non-rotated files ? Why not just take the .log files without their rotation ?

[monitor:///data/production/logs/JobBatch/documents/*.log]
disabled = false
index = logs_jobbatch
sourcetype = JobBatch

Cheers,
David

0 Karma

arun_kant_sharm
Path Finder

Hi @DavidHourani

Thanks for your reply. That location(path) have many other files, so I require whitelist, because I want to monitor both .log.gz and .log file therefore I give whitelist = (zipDocument_[\d]+.log.gz|zipDocument_[\d]+.log)

0 Karma

DavidHourani
Super Champion

oh, okay so you need both, that would duplicate your logs though since the .log and the .log.gz file are the same no -- just rotated 1 day later ?

In any case use crcSalt if thats what you're trying to achieve :

[monitor:///data/production/logs/JobBatch/documents]
disabled = false
index = logs_jobbatch
sourcetype = JobBatch
whitelist = zipDocument_[\d]+(\.log.gz|\.log)$
crcSalt = <SOURCE>
0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!