Windows monitored file duplicate data

darrenfuller — Tue, 23 Mar 2021 20:17:23 GMT

I have a csv file that is written to once a day.

The input points to a custom sourcetype [csvtest] which has appropriate settings for the data within.

inputs.conf:

[monitor://c:\opt\splunk\etc\apps\csvtest\data\csvtest.csv] index = main sourcetype = csvtest

props.conf:

[csvtest] LINE_BREAKER = ([\r\n]+) SHOULD_LINEMERGE = false TIME_PREFIX = ^ TIME_FORMAT = %Y-%m-%d MAX_TIMESTAMP_LOOKAHEAD = 15 TRUNCATE = 10000

and data looks as so:

2020-01-26,,,1.0,,,,,,,,9,,,,,,,,,, 2020-01-27,,,2.0,,,,,,,,19,,,,,,,,,, 2020-01-28,,,1.0,1.0,,,1.0,,,,11,,,,,,,,,, 2020-01-30,,,0.0,2.0,,,2.0,,,,27,,,,,,,,,, 2020-01-31,,,0.0,2.0,,,2.0,,,,17,,,,,,,,,, 2020-02-03,,,0.0,3.0,,,3.0,,,,29,,,,,,,,,, 2020-02-04,90.0,12.0,0.0,3.0,,,3.0,139.0,,,34,,,,,,,,,, 2020-02-05,96.0,8.0,0.0,3.0,,,3.0,150.0,,,43,,,,,,,,,, 2020-02-06,104.0,0.0,0.0,3.0,,,3.0,169.0,,,62,,,,,,,,,, 2020-02-08,130.0,25.0,0.0,3.0,,,3.0,197.0,,,39,,,,,,,,,, 2020-02-10,167.0,81.0,0.0,3.0,,,3.0,259.0,,,8,,,,,,,,,, 2020-02-11,184.0,79.0,0.0,3.0,,,3.0,285.0,,,19,,,,,,,,,, 2020-02-12,257.0,44.0,0.0,2.0,1.0,,3.0,313.0,,,9,,,,,,,,,, 2020-02-13,306.0,16.0,0.0,2.0,1.0,,3.0,340.0,,,15,,,,,,,,,, 2020-02-14,353.0,0.0,0.0,2.0,1.0,,3.0,364.0,,,8,,,,,,,,,, 2020-02-17,399.0,0.0,0.0,2.0,1.0,,3.0,402.0,,,0,,,,,,,,,, 2020-02-18,418.0,0.0,0.0,2.0,1.0,,3.0,421.0,,,0,,,,,,,,,, 2020-02-19,436.0,0.0,0.0,2.0,1.0,,3.0,456.0,,,17,,,,,,,,,, 2020-02-20,462.0,,0.0,1.0,2.0,,3.0,479.0,,,14,,,,,,,,,, 2020-02-21,483.0,,0.0,0.0,3.0,,3.0,498.0,,,12,,,,,,,,,, 2020-02-22,540.0,,0.0,1.0,3.0,,4.0,553.0,,,9,,,,,,,,,,

Every time this file adds a new line to the bottom , the whole file is ingested... causing duplicate data in the index.

I have created the same scenario on Linux and the forwarder appropriately uses the default initCrcLength and identifies it has seen the file before and only ingests the new event.

in the _internal index, i am seeing events like so>

03-23-2021 20:00:02.851 -0400 INFO WatchedFile - Will begin reading at offset=0 for file='C:\opt\splunk\etc\apps\csvtest\data\csvtest.csv'.

Is this just a windows forwarder symptom? Is there an inputs.conf setting I can utilize to fix this?

Thanks in advance!

topic Windows monitored file duplicate data in Getting Data In

Windows monitored file duplicate data