topic Reindex Duplicates / Reindex duplicate data in Getting Data In

Reindex Duplicates / Reindex duplicate data

omuelle1 — Wed, 18 Jan 2017 14:34:30 GMT

HI Splunkers,

I got a little complicated issue I cannot figure out.

Everyday I receive a host file that we index that contains 80% of that data is duplicate from the days before (same timestamp everything). Splunk goes ahead and only indexes the non duplicates, which I would usually appreciate.

However, for a specific report I need to find a setting where Splunk overwrites the old events and indexes a duplicate from the latest source.

For example:

I get event 1111 - timestamp 1-17  on 1/17 in logfile1_17.txt

on 1-18 I have the event 1111 - timestamp 1-17 in logfile1_18.txt 

Now when I look for this event 1111 , I find as source logfile1_17.txt but I would need it indexed again and have source logfile1_18.txt

Is there a setting that the event 1111 is shown in splunk with the latest indexed file as source or is indexed again ?

Thanks.

Oliver

Re: Reindex Duplicates / Reindex duplicate data

nabeel652 — Thu, 19 Jan 2017 04:40:36 GMT

If this is the behaviour you want then consider using batch instead of monitor.

[batch://<path>]
disabled = false
move_policy = sinkhole
index = yourindex
sourcetype = somesourcetype

This will index any file put in that directory and delete it once ingested (so keep that in mind that you may lose the file once indexed). However the move_policy set to sinkhole will reindex the same file if put in that folder with different timestamp instead of following the tail.

Re: Reindex Duplicates / Reindex duplicate data

omuelle1 — Thu, 19 Jan 2017 15:30:40 GMT

Thank you, I went crcSalt = which also seems to reindex duplicates.

Re: Reindex Duplicates / Reindex duplicate data

gurugv — Tue, 21 Mar 2017 15:46:33 GMT

Would this duplicate existing data? Would the summary indices be automatically updated?