Getting Data In

Windows monitored file duplicate data

darrenfuller
Contributor

I have a csv file that is written to once a day.   

The input points to a custom sourcetype [csvtest] which has appropriate settings for the data within.  

inputs.conf:

 

[monitor://c:\opt\splunk\etc\apps\csvtest\data\csvtest.csv]
index = main 
sourcetype = csvtest

 

 

props.conf:

 

[csvtest]
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = false
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d
MAX_TIMESTAMP_LOOKAHEAD = 15
TRUNCATE = 10000

 

 

and data looks as so: 

 

2020-01-26,,,1.0,,,,,,,,9,,,,,,,,,,
2020-01-27,,,2.0,,,,,,,,19,,,,,,,,,,
2020-01-28,,,1.0,1.0,,,1.0,,,,11,,,,,,,,,,
2020-01-30,,,0.0,2.0,,,2.0,,,,27,,,,,,,,,,
2020-01-31,,,0.0,2.0,,,2.0,,,,17,,,,,,,,,,
2020-02-03,,,0.0,3.0,,,3.0,,,,29,,,,,,,,,,
2020-02-04,90.0,12.0,0.0,3.0,,,3.0,139.0,,,34,,,,,,,,,,
2020-02-05,96.0,8.0,0.0,3.0,,,3.0,150.0,,,43,,,,,,,,,,
2020-02-06,104.0,0.0,0.0,3.0,,,3.0,169.0,,,62,,,,,,,,,,
2020-02-08,130.0,25.0,0.0,3.0,,,3.0,197.0,,,39,,,,,,,,,,
2020-02-10,167.0,81.0,0.0,3.0,,,3.0,259.0,,,8,,,,,,,,,,
2020-02-11,184.0,79.0,0.0,3.0,,,3.0,285.0,,,19,,,,,,,,,,
2020-02-12,257.0,44.0,0.0,2.0,1.0,,3.0,313.0,,,9,,,,,,,,,,
2020-02-13,306.0,16.0,0.0,2.0,1.0,,3.0,340.0,,,15,,,,,,,,,,
2020-02-14,353.0,0.0,0.0,2.0,1.0,,3.0,364.0,,,8,,,,,,,,,,
2020-02-17,399.0,0.0,0.0,2.0,1.0,,3.0,402.0,,,0,,,,,,,,,,
2020-02-18,418.0,0.0,0.0,2.0,1.0,,3.0,421.0,,,0,,,,,,,,,,
2020-02-19,436.0,0.0,0.0,2.0,1.0,,3.0,456.0,,,17,,,,,,,,,,
2020-02-20,462.0,,0.0,1.0,2.0,,3.0,479.0,,,14,,,,,,,,,,
2020-02-21,483.0,,0.0,0.0,3.0,,3.0,498.0,,,12,,,,,,,,,,
2020-02-22,540.0,,0.0,1.0,3.0,,4.0,553.0,,,9,,,,,,,,,,

 

 

Every time this file adds a new line to the bottom , the whole file is ingested... causing duplicate data in the index.

I have created the same scenario on Linux and the forwarder appropriately uses the default initCrcLength and identifies it has seen the file before and only ingests the new event.   

in the _internal index, i am seeing events like so> 

 

03-23-2021 20:00:02.851 -0400 INFO  WatchedFile - Will begin reading at offset=0 for file='C:\opt\splunk\etc\apps\csvtest\data\csvtest.csv'.

 

 

Is this just a windows forwarder symptom?    Is there an inputs.conf setting I can utilize to fix this? 

Thanks in advance!

Labels (3)
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...