Getting Data In
Highlighted

Event Count keeps increasing when monitoring CSV file

New Member

I have configured a CSV file path using Monitor files and directories option in the Add Data feature. That CSV file having 1,20,742 records(events). But when doing search in splunk, this event count is keep on increasing. I have inserted 6 records into that csv file. those records have been displayed in the splunk search. But the problem is event count. Now it shows 8,45,934 events. How is it possible since the source file having only 1,20,748 records and why the event count is keep on increasing.

Even after removing all the pipes(|) from the query, its showing the 8,45,934 only. How to avoid this problem?

0 Karma
Highlighted

Re: Event Count keeps increasing when monitoring CSV file

SplunkTrust
SplunkTrust

Please share the inputs.conf stanza for that file.

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Highlighted

Re: Event Count keeps increasing when monitoring CSV file

New Member

inputs.conf file below:

[tcp://443]
connectionhost = dns
index = main
sourcetype = syslog
[WinHostMon://MyMachine]
index = main
interval = 1800
type = Roles;NetworkAdapter;Service;OperatingSystem;Driver;Processor;Disk;Computer;Process
[monitor://C:...\Documents\Talend\APM\OSH
Data\out-apmtsaug31st.csv]
disabled = false
index = mnd
osh
sourcetype = oshtscsv

FYI: i have not updated this file when configure monitoring file. I just used the UI option to configure these settings and opted the "Continously Monitor" option.

0 Karma
Highlighted

Re: Event Count keeps increasing when monitoring CSV file

SplunkTrust
SplunkTrust

Please provide a sample of the csv data and your props.conf as well. I believe your line breaking is off.

0 Karma
Highlighted

Re: Event Count keeps increasing when monitoring CSV file

New Member

pls find below the sample records in the csv file:

123456.ABC,2017-09-01T00:00:00.000Z,1,2
123457.ABC,2017-09-05T00:00:00.000Z,2,2
123458.ABC,2017-08-01T00:00:00.000Z,0,3
123459.ABC,2017-08-01T00:05:00.000Z,0,3
123460.ABC,2017-08-01T00:10:00.000Z,0,3
123461.ABC,2017-08-01T00:15:00.000Z,0,3
123462.ABC,2017-08-01T00:20:00.000Z,0,3
123463.ABC,2017-08-01T00:25:00.000Z,0,3

props.conf file:

[oshtscsv]
DATETIMECONFIG =
INDEXED
EXTRACTIONS = csv
KVMODE = none
NO
BINARYCHECK = true
SHOULD
LINEMERGE = false
category = Structured
description = Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled = false
pulldowntype = true
FIELD
NAMES = resourcetag, timestamp, value, quality
TIMESTAMP
FIELDS = timestamp
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N

0 Karma
Highlighted

Re: Event Count keeps increasing when monitoring CSV file

SplunkTrust
SplunkTrust

TIME_FORMAT should be %Y-%m-%dT%H:%M:%S.%3N%Z
The other settings look OK to me.

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Highlighted

Re: Event Count keeps increasing when monitoring CSV file

New Member

TIME_FORMAT given as you mentioned %Y-%m-%dT%H:%M:%S.%3N%Z.
Able to do search and getting results. the only problem is EventCount is keep on increasing.
EventCount should always equal to the records/lines in the source file. But it increased 7 times.

0 Karma
Highlighted

Re: Event Count keeps increasing when monitoring CSV file

Esteemed Legend

My suspicion is that you are replacing the entire file, not adding to it with something like echo "This is a test" >> MyLogFile. Try a proper test using something that actually adds to the bottom of the file instead of something that replaces the entire file with the same stuff plus some other stuff. It is your test methodology that is broken, not the file or Splunk.

0 Karma