Splunk Search

File indexed only occasionally

MuratKuru
Explorer

My input.conf file:
[monitor:///var/log/openvpn/hostname_vpnStatus.log]
disabled = 0
crcSalt = SOURCE
index = iss-nipa-clients
sourcetype = nipa:clients:status

My props.conf file:
[nipa:clients:status]
[source::/var/log/openvpn/hostname_vpnStatus.log]
CHECK_METHOD = modtime
DATETIME_CONFIG = NONE

Extract from the forwarder splunkd.log:
09-13-2017 11:55:02.104 +0200 INFO WatchedFile - Modtime is newer than stored, will reread file='/var/log/openvpn/hostname_vpnStatus.log'.
09-13-2017 11:55:02.110 +0200 INFO WatchedFile - Will begin reading at offset=0 for file='/var/log/openvpn/hostname_vpnStatus.log'.

The file to be indexed:
File created at: 2017-09-13_11:59:01
UNDEF,ip.ip.ip.ip:port,84,188,Wed Sep 13 11:58:16 2017,Tunnel_a
c1115-ip.ip.ip.ip:port,19051077,18985566,Thu Aug 31 14:54:56 2017,Tunnel_a
c1350,ip.ip.ip.ip:port,161253,160644,Wed Sep 13 09:24:57 2017,Tunnel_a
c1255-1,ip.ip.ip.ip:port,176571,172050,Wed Sep 13 09:24:57 2017,Tunnel_a
c1783-1,ip.ip.ip.ip:port,170017,175415,Wed Sep 13 09:24:59 2017,Tunnel_d
c1215-1,ip.ip.ip.ip:port,167136,167643,Wed Sep 13 09:24:56 2017,Tunnel_d
File created at: 2017-09-13_11:59:01

This file is created every minute and according to splunkd.log it is also read every minute, but not indexed only periodicaly.
The created time stamp on the header and trailer is changing every minute as the creatation time of the file.

Why is splunk not indexing this file every minute!!!!????

Tags (1)
0 Karma

gcusello
Esteemed Legend

Hi MuratKuru,
at first why you have two stanzas in props.conf? the first one is empty and only the second one is read; try to use only sourcetype stanza ( [nipa:clients:status] ) in your props.conf.
In addition if you have crcSalt = <SOURCE> and you index always the same file, you could have problems.
So try to modify inputs.conf and props.conf and check if the situation is the same or not.
Bye.
Giuseppe

0 Karma

MuratKuru
Explorer

Hi Giuseppe
I have made the suggested changes, but still have the save problem.
Regards,
Murat Kuru

0 Karma

gcusello
Esteemed Legend

Hi Murat Kuru,
Is there a reason because you perform a File checksum configuration using CHECK_METHOD = modtime ?
Try to cut this option.
bye.
Giuseppe

0 Karma

ddrillic
Ultra Champion

I agree with @cusello. The following speaks about a similar issue - Log file in /etc/log is reindexed resulting in duplicate events

CHECK_METHOD = modtime seems there to be the culprit...

0 Karma

gcusello
Esteemed Legend

If this answer satisfies your question, please accept or upvote it.
Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...

Announcing General Availability of Splunk Incident Intelligence!

Digital transformation is real! Across industries, companies big and small are going through rapid digital ...