topic Re: Event Count keeps increasing when monitoring CSV file in Getting Data In

Event Count keeps increasing when monitoring CSV file

tskarthic — Wed, 06 Sep 2017 13:33:31 GMT

I have configured a CSV file path using Monitor files and directories option in the Add Data feature. That CSV file having 1,20,742 records(events). But when doing search in splunk, this event count is keep on increasing. I have inserted 6 records into that csv file. those records have been displayed in the splunk search. But the problem is event count. Now it shows 8,45,934 events. How is it possible since the source file having only 1,20,748 records and why the event count is keep on increasing.

Even after removing all the pipes(|) from the query, its showing the 8,45,934 only. How to avoid this problem?

Re: Event Count keeps increasing when monitoring CSV file

richgalloway — Wed, 06 Sep 2017 14:02:04 GMT

Please share the inputs.conf stanza for that file.

Re: Event Count keeps increasing when monitoring CSV file

tskarthic — Tue, 29 Sep 2020 15:39:35 GMT

inputs.conf file below:

[tcp://443]
connection_host = dns
index = main
sourcetype = syslog
[WinHostMon://MyMachine]
index = main
interval = 1800
type = Roles;NetworkAdapter;Service;OperatingSystem;Driver;Processor;Disk;Computer;Process
[monitor://C:...\Documents\Talend\APM\OSH_Data\out-apmts_aug31st.csv]
disabled = false
index = mnd_osh
sourcetype = osh_ts_csv

FYI: i have not updated this file when configure monitoring file. I just used the UI option to configure these settings and opted the "Continously Monitor" option.

Re: Event Count keeps increasing when monitoring CSV file

jkat54 — Wed, 06 Sep 2017 15:06:11 GMT

Please provide a sample of the csv data and your props.conf as well. I believe your line breaking is off.

Re: Event Count keeps increasing when monitoring CSV file

tskarthic — Tue, 29 Sep 2020 15:39:41 GMT

pls find below the sample records in the csv file:

123456.ABC,2017-09-01T00:00:00.000Z,1,2
123457.ABC,2017-09-05T00:00:00.000Z,2,2
123458.ABC,2017-08-01T00:00:00.000Z,0,3
123459.ABC,2017-08-01T00:05:00.000Z,0,3
123460.ABC,2017-08-01T00:10:00.000Z,0,3
123461.ABC,2017-08-01T00:15:00.000Z,0,3
123462.ABC,2017-08-01T00:20:00.000Z,0,3
123463.ABC,2017-08-01T00:25:00.000Z,0,3

props.conf file:

[osh_ts_csv]
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = csv
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Structured
description = Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled = false
pulldown_type = true
FIELD_NAMES = resource_tag, timestamp, value, quality
TIMESTAMP_FIELDS = timestamp
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N

Re: Event Count keeps increasing when monitoring CSV file

richgalloway — Wed, 06 Sep 2017 17:12:13 GMT

TIME_FORMAT should be %Y-%m-%dT%H:%M:%S.%3N%Z
The other settings look OK to me.

Re: Event Count keeps increasing when monitoring CSV file

tskarthic — Wed, 06 Sep 2017 17:45:04 GMT

TIME_FORMAT given as you mentioned %Y-%m-%dT%H:%M:%S.%3N%Z.
Able to do search and getting results. the only problem is EventCount is keep on increasing.
EventCount should always equal to the records/lines in the source file. But it increased 7 times.

Re: Event Count keeps increasing when monitoring CSV file

woodcock — Sun, 10 Sep 2017 05:18:51 GMT

My suspicion is that you are replacing the entire file, not adding to it with something like echo "This is a test" >> MyLogFile. Try a proper test using something that actually adds to the bottom of the file instead of something that replaces the entire file with the same stuff plus some other stuff. It is your test methodology that is broken, not the file or Splunk.