I have a csv file that is written to once a day.
The input points to a custom sourcetype [csvtest] which has appropriate settings for the data within.
inputs.conf:
[monitor://c:\opt\splunk\etc\apps\csvtest\data\csvtest.csv]
index = main
sourcetype = csvtest
props.conf:
[csvtest]
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = false
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d
MAX_TIMESTAMP_LOOKAHEAD = 15
TRUNCATE = 10000
and data looks as so:
2020-01-26,,,1.0,,,,,,,,9,,,,,,,,,,
2020-01-27,,,2.0,,,,,,,,19,,,,,,,,,,
2020-01-28,,,1.0,1.0,,,1.0,,,,11,,,,,,,,,,
2020-01-30,,,0.0,2.0,,,2.0,,,,27,,,,,,,,,,
2020-01-31,,,0.0,2.0,,,2.0,,,,17,,,,,,,,,,
2020-02-03,,,0.0,3.0,,,3.0,,,,29,,,,,,,,,,
2020-02-04,90.0,12.0,0.0,3.0,,,3.0,139.0,,,34,,,,,,,,,,
2020-02-05,96.0,8.0,0.0,3.0,,,3.0,150.0,,,43,,,,,,,,,,
2020-02-06,104.0,0.0,0.0,3.0,,,3.0,169.0,,,62,,,,,,,,,,
2020-02-08,130.0,25.0,0.0,3.0,,,3.0,197.0,,,39,,,,,,,,,,
2020-02-10,167.0,81.0,0.0,3.0,,,3.0,259.0,,,8,,,,,,,,,,
2020-02-11,184.0,79.0,0.0,3.0,,,3.0,285.0,,,19,,,,,,,,,,
2020-02-12,257.0,44.0,0.0,2.0,1.0,,3.0,313.0,,,9,,,,,,,,,,
2020-02-13,306.0,16.0,0.0,2.0,1.0,,3.0,340.0,,,15,,,,,,,,,,
2020-02-14,353.0,0.0,0.0,2.0,1.0,,3.0,364.0,,,8,,,,,,,,,,
2020-02-17,399.0,0.0,0.0,2.0,1.0,,3.0,402.0,,,0,,,,,,,,,,
2020-02-18,418.0,0.0,0.0,2.0,1.0,,3.0,421.0,,,0,,,,,,,,,,
2020-02-19,436.0,0.0,0.0,2.0,1.0,,3.0,456.0,,,17,,,,,,,,,,
2020-02-20,462.0,,0.0,1.0,2.0,,3.0,479.0,,,14,,,,,,,,,,
2020-02-21,483.0,,0.0,0.0,3.0,,3.0,498.0,,,12,,,,,,,,,,
2020-02-22,540.0,,0.0,1.0,3.0,,4.0,553.0,,,9,,,,,,,,,,
Every time this file adds a new line to the bottom , the whole file is ingested... causing duplicate data in the index.
I have created the same scenario on Linux and the forwarder appropriately uses the default initCrcLength and identifies it has seen the file before and only ingests the new event.
in the _internal index, i am seeing events like so>
03-23-2021 20:00:02.851 -0400 INFO WatchedFile - Will begin reading at offset=0 for file='C:\opt\splunk\etc\apps\csvtest\data\csvtest.csv'.
Is this just a windows forwarder symptom? Is there an inputs.conf setting I can utilize to fix this?
Thanks in advance!
