All Apps and Add-ons

Issue with CSV File monitoring on Universal Forwarder

thirusama
Path Finder

Splunk Version 6.3.4

We are monitoring a csv file with same name which gets overwritten/updated in every 30 minutes. The issue seems to be intermittent i.e. it picks up the file sometime & sometimes it does not. I tried changing options like "initCrcLength" with 1024, 10240 & 1048575. None of them helped.

Since 2-3 days, I am seeing that it is reading only one line, that too partial line from the file. I have set up the inputs.conf & props.conf on Forwarder (deployed thru deployment server). Here are the current settings & the error I am getting.

inputs.conf
[monitor://C:\Temp\incident*.csv]
disabled = 0
sourcetype = imdp:ITSM:incidents_new
index = imdc_w
crcSalt = SOURCE ( with less than & greate than also included)
initCrcLength = 1048575
ignoreOlderThan = 14d

alwaysOpenFile = 1

time_before_close =15

props.conf
[imdp:ITSM:incidents_new]
INDEXED_EXTRACTIONS = csv
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Structured
description = Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled = false
pulldown_type = true
MAX_DAYS_AGO = 1000
MAX_DAYS_HENCE = 60
TIMESTAMP_FIELDS="opened_at"

HEADER_FIELD_LINE_NUMBER=1

[source::C:\Temp\incident.csv]
CHECK_METHOD = modtime

Here is how my sample file looks like

"number","incident_state","assignment_group","caller_id","opened_at","u_incident_assigned","u_im_service_restored_date_tim","short_description","u_im_sla_breached","severity","u_im_reporter_grp","u_im_caller_city","assigned_to","u_axp_im_config_item","u_axp_im_closureci","caused_by","u_im_causefaultychg"
"INC0000000","New","AXPVO_ABCDL","abc_name","09-23-2016 09:36:37","09-23-2016 10:00:05","","description-sample","false","Sev4","LAIBMHD_group","D.F.","","Avaya Voice","Avaya Voice","",""
----------so on----

------------But splunk did not pick any of the lines-- but just picked some intermediate line & that too half of the line..

pabc516 08:04:47 Password validation for user abc failed","false","Sev5","NGIDBA_def_AM","","name Ramadoss","APDWD516","APDWD517","",""

---- I do not see any issue with timestamp in the file for any of the rows.

This is what I see in splunkd.log.

09-23-2016 10:03:13.148 -0700 WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Fri Sep 23 06:13:07 2016). Context: source::C:\Temp\incident.csv|host::WGPIS850|imdp:ITSM:incidents_new|673
host = WGPIS850 source = C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log sourcetype = splunkd

09-23-2016 10:03:13.132 -0700 INFO WatchedFile - Resetting fd to re-extract header.
host = WGPIS850 source = C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log sourcetype = splunkd

09-23-2016 10:03:13.132 -0700 INFO WatchedFile - Will begin reading at offset=0 for file='C:\Temp\incident.csv'.
host = WGPIS850 source = C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log sourcetype = splunkd

09-23-2016 10:03:13.132 -0700 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='C:\Temp\incident.csv'.

0 Karma
1 Solution

thirusama
Path Finder

This issue is resolved. We were using old version of SPlunk forwarder. Now we have upgraded it to Splunk 6.3.4

View solution in original post

0 Karma

thirusama
Path Finder

This issue is resolved. We were using old version of SPlunk forwarder. Now we have upgraded it to Splunk 6.3.4

0 Karma
Get Updates on the Splunk Community!

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...