All Apps and Add-ons

Splunk not ingest most recent logs.

arun_kant_sharm
Path Finder

Hi Experts,

I try to ingest data from a mount point, in that path all files are log rotate and moved to .gz format on daily basis.
But my splunk ingest only that file that already converted into .gz file. So before 24 hours I don't find latest logs.
Please suggest how i get latest logs in the Splunk.

Sample conf files are :
inputs.conf :
[monitor:///data/production/logs/JobBatch/documents]
disabled = false
index = logs_jobbatch
sourcetype = JobBatch
whitelist = (zipDocument_[\d]+.log.gz|zipDocument_[\d]+.log)

props.conf :
[JobBatch]
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = false
category = Custom
disabled = false
pulldown_type = true
BREAK_ONLY_BEFORE = ^\d{4}-\d{2}-\d{2}\s{1}\d{2}:\d{2}:\d{2}.\d{3}\s{1}
HEADER_FIELD_LINE_NUMBER = 9
TRUNCATE = 750000
MAX_EVENTS = 1000
BREAK_ONLY_BEFORE_DATE =
SHOULD_LINEMERGE = false

0 Karma

DavidHourani
Super Champion

Hi @arun_kant_sharma,

You're ingesting both rotated files and non-rotated files ? Why not just take the .log files without their rotation ?

[monitor:///data/production/logs/JobBatch/documents/*.log]
disabled = false
index = logs_jobbatch
sourcetype = JobBatch

Cheers,
David

0 Karma

arun_kant_sharm
Path Finder

Hi @DavidHourani

Thanks for your reply. That location(path) have many other files, so I require whitelist, because I want to monitor both .log.gz and .log file therefore I give whitelist = (zipDocument_[\d]+.log.gz|zipDocument_[\d]+.log)

0 Karma

DavidHourani
Super Champion

oh, okay so you need both, that would duplicate your logs though since the .log and the .log.gz file are the same no -- just rotated 1 day later ?

In any case use crcSalt if thats what you're trying to achieve :

[monitor:///data/production/logs/JobBatch/documents]
disabled = false
index = logs_jobbatch
sourcetype = JobBatch
whitelist = zipDocument_[\d]+(\.log.gz|\.log)$
crcSalt = <SOURCE>
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...