Splunk not ingest most recent logs.

[arun_kant_sharm]

Hi Experts,

I try to ingest data from a mount point, in that path all files are log rotate and moved to .gz format on daily basis.
But my splunk ingest only that file that already converted into .gz file. So before 24 hours I don't find latest logs.
Please suggest how i get latest logs in the Splunk.

Sample conf files are :
inputs.conf :
[monitor:///data/production/logs/JobBatch/documents]
disabled = false
index = logs_jobbatch
sourcetype = JobBatch
whitelist = (zipDocument_[\d]+.log.gz|zipDocument_[\d]+.log)

props.conf :
[JobBatch]
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = false
category = Custom
disabled = false
pulldown_type = true
BREAK_ONLY_BEFORE = ^\d{4}-\d{2}-\d{2}\s{1}\d{2}:\d{2}:\d{2}.\d{3}\s{1}
HEADER_FIELD_LINE_NUMBER = 9
TRUNCATE = 750000
MAX_EVENTS = 1000
BREAK_ONLY_BEFORE_DATE =
SHOULD_LINEMERGE = false

[DavidHourani]

Hi @arun_kant_sharma,

You're ingesting both rotated files and non-rotated files ? Why not just take the .log files without their rotation ?

[monitor:///data/production/logs/JobBatch/documents/*.log]
disabled = false
index = logs_jobbatch
sourcetype = JobBatch

Cheers,
David

[arun_kant_sharm]

Hi @DavidHourani

Thanks for your reply. That location(path) have many other files, so I require whitelist, because I want to monitor both .log.gz and .log file therefore I give whitelist = (zipDocument_[\d]+.log.gz|zipDocument_[\d]+.log)

[DavidHourani]

oh, okay so you need both, that would duplicate your logs though since the .log and the .log.gz file are the same no -- just rotated 1 day later ?

In any case use crcSalt if thats what you're trying to achieve :

[monitor:///data/production/logs/JobBatch/documents]
disabled = false
index = logs_jobbatch
sourcetype = JobBatch
whitelist = zipDocument_[\d]+(\.log.gz|\.log)$
crcSalt = <SOURCE>