Splunk Search

File indexed only occasionally

MuratKuru
Explorer

My input.conf file:
[monitor:///var/log/openvpn/hostname_vpnStatus.log]
disabled = 0
crcSalt = SOURCE
index = iss-nipa-clients
sourcetype = nipa:clients:status

My props.conf file:
[nipa:clients:status]
[source::/var/log/openvpn/hostname_vpnStatus.log]
CHECK_METHOD = modtime
DATETIME_CONFIG = NONE

Extract from the forwarder splunkd.log:
09-13-2017 11:55:02.104 +0200 INFO WatchedFile - Modtime is newer than stored, will reread file='/var/log/openvpn/hostname_vpnStatus.log'.
09-13-2017 11:55:02.110 +0200 INFO WatchedFile - Will begin reading at offset=0 for file='/var/log/openvpn/hostname_vpnStatus.log'.

The file to be indexed:
File created at: 2017-09-13_11:59:01
UNDEF,ip.ip.ip.ip:port,84,188,Wed Sep 13 11:58:16 2017,Tunnel_a
c1115-ip.ip.ip.ip:port,19051077,18985566,Thu Aug 31 14:54:56 2017,Tunnel_a
c1350,ip.ip.ip.ip:port,161253,160644,Wed Sep 13 09:24:57 2017,Tunnel_a
c1255-1,ip.ip.ip.ip:port,176571,172050,Wed Sep 13 09:24:57 2017,Tunnel_a
c1783-1,ip.ip.ip.ip:port,170017,175415,Wed Sep 13 09:24:59 2017,Tunnel_d
c1215-1,ip.ip.ip.ip:port,167136,167643,Wed Sep 13 09:24:56 2017,Tunnel_d
File created at: 2017-09-13_11:59:01

This file is created every minute and according to splunkd.log it is also read every minute, but not indexed only periodicaly.
The created time stamp on the header and trailer is changing every minute as the creatation time of the file.

Why is splunk not indexing this file every minute!!!!????

Tags (1)
0 Karma

gcusello
Esteemed Legend

Hi MuratKuru,
at first why you have two stanzas in props.conf? the first one is empty and only the second one is read; try to use only sourcetype stanza ( [nipa:clients:status] ) in your props.conf.
In addition if you have crcSalt = <SOURCE> and you index always the same file, you could have problems.
So try to modify inputs.conf and props.conf and check if the situation is the same or not.
Bye.
Giuseppe

0 Karma

MuratKuru
Explorer

Hi Giuseppe
I have made the suggested changes, but still have the save problem.
Regards,
Murat Kuru

0 Karma

gcusello
Esteemed Legend

Hi Murat Kuru,
Is there a reason because you perform a File checksum configuration using CHECK_METHOD = modtime ?
Try to cut this option.
bye.
Giuseppe

0 Karma

ddrillic
Ultra Champion

I agree with @cusello. The following speaks about a similar issue - Log file in /etc/log is reindexed resulting in duplicate events

CHECK_METHOD = modtime seems there to be the culprit...

0 Karma

gcusello
Esteemed Legend

If this answer satisfies your question, please accept or upvote it.
Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Announcing General Availability of Splunk Incident Intelligence!

Digital transformation is real! Across industries, companies big and small are going through rapid digital ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...