https://community.splunk.com/t5/Getting-Data-In/How-to-handle-a-daily-changing-CSV-file-and-avoid-indexing/m-p/211875#M41712 <P>If below is your case:</P> <P>Sep,1,2015:<BR /> If you have ten records.<BR /> Sep,2,2015:<BR /> If you have ten+7 records. (7 new records and 10 records regenerated and are same as the previous file had)<BR /> Sep,3,2015:<BR /> You have 25 records in total. (10 from Sep,1,2015. 7 from Sep,2,2015 and 8 new records from Sep,3,2015)</P> <P>Then, you can choose to monitor the file continuously (while indexing) and make sure you copy paste all the data from the new file into old file (if you want to do it manually). This way you don't have duplicate records.</P> <P>If your scenario is different, then just use <CODE>index=something sourcetype=csv source=path/filename.csv | dedup _raw | your analysis code</CODE></P> <P>Hope this is helpful for you.</P> Wed, 02 Sep 2015 13:55:34 GMT thirumalreddyb 2015-09-02T13:55:34Z https://community.splunk.com/t5/Getting-Data-In/How-to-handle-a-daily-changing-CSV-file-and-avoid-indexing/m-p/211874#M41711 <P>Hi,</P> <P>I have daily growing CSV file that I want to index. Just importing it every day would end up in a lot a duplicate events. I've read about the followTail option, but also that this option is not recommended. How I can avoid duplicate events? My first thought was to create a daily scheduled search to delete all "old" files and only keep the last indexed file, but I hope there is a better possibility.</P> <P>Cheers<BR /> Heinz</P> Wed, 02 Sep 2015 10:04:36 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-handle-a-daily-changing-CSV-file-and-avoid-indexing/m-p/211874#M41711 HeinzWaescher 2015-09-02T10:04:36Z https://community.splunk.com/t5/Getting-Data-In/How-to-handle-a-daily-changing-CSV-file-and-avoid-indexing/m-p/211875#M41712 <P>If below is your case:</P> <P>Sep,1,2015:<BR /> If you have ten records.<BR /> Sep,2,2015:<BR /> If you have ten+7 records. (7 new records and 10 records regenerated and are same as the previous file had)<BR /> Sep,3,2015:<BR /> You have 25 records in total. (10 from Sep,1,2015. 7 from Sep,2,2015 and 8 new records from Sep,3,2015)</P> <P>Then, you can choose to monitor the file continuously (while indexing) and make sure you copy paste all the data from the new file into old file (if you want to do it manually). This way you don't have duplicate records.</P> <P>If your scenario is different, then just use <CODE>index=something sourcetype=csv source=path/filename.csv | dedup _raw | your analysis code</CODE></P> <P>Hope this is helpful for you.</P> Wed, 02 Sep 2015 13:55:34 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-handle-a-daily-changing-CSV-file-and-avoid-indexing/m-p/211875#M41712 thirumalreddyb 2015-09-02T13:55:34Z https://community.splunk.com/t5/Getting-Data-In/How-to-handle-a-daily-changing-CSV-file-and-avoid-indexing/m-p/211876#M41713 <P>Maybe you should consider looking at the kv store. I believe it has an upsert capability through a RESTful interface.</P> <P><A href="http://docs.splunk.com/Documentation/Splunk/6.2.5/Admin/AboutKVstore">http://docs.splunk.com/Documentation/Splunk/6.2.5/Admin/AboutKVstore</A></P> Wed, 02 Sep 2015 17:34:49 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-handle-a-daily-changing-CSV-file-and-avoid-indexing/m-p/211876#M41713 jaredlaney 2015-09-02T17:34:49Z https://community.splunk.com/t5/Getting-Data-In/How-to-handle-a-daily-changing-CSV-file-and-avoid-indexing/m-p/211877#M41714 <P>| dedup _raw is good a first workaround</P> <P>thanks!</P> Mon, 14 Sep 2015 08:33:46 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-handle-a-daily-changing-CSV-file-and-avoid-indexing/m-p/211877#M41714 HeinzWaescher 2015-09-14T08:33:46Z