topic Re: is it possible to run remove duplicates on specific dates and not on all time? in Splunk Search

is it possible to run remove duplicates on specific dates and not on all time?

gooza — Tue, 19 Jul 2011 07:07:12 GMT

When using the remove duplicate event python is it possible to run it on specific date range?

Re: is it possible to run remove duplicates on specific dates and not on all time?

gooza — Tue, 19 Jul 2011 07:24:56 GMT

Found it..
Edited the script and added to line 56 (starting with search1) earliest= latest dates :
"search1 = 'search index=' + index + 'earliest=129461760 latest=1294704000 | ..."

I don't know Perl , anyone knows how to add these dates as parameters ?

Re: is it possible to run remove duplicates on specific dates and not on all time?

zpavic — Tue, 19 Jul 2011 17:42:10 GMT

I create new script for you. http://pastebin.com/JusgWRMy

Usage: ./splunk cmd python rem_dupl_event_spec_date.py <index> <earliest_date> <latest_date>
earliest_date & latest_date syntax: %m/%d/%Y:%H:%M:%S

For example:

./splunk cmd python rem_dupl_event_spec_date.py test 05/20/2011:0:0:0 05/24/2011:23:59:59

Re: is it possible to run remove duplicates on specific dates and not on all time?

gooza — Wed, 20 Jul 2011 06:59:07 GMT

works great , Thanks!

Re: is it possible to run remove duplicates on specific dates and not on all time?

kphillipson — Wed, 07 Nov 2012 20:54:24 GMT

Really like this. Is there a way for it to only look at a source or sourcetype within a given index?

Re: is it possible to run remove duplicates on specific dates and not on all time?

karthy — Fri, 16 Nov 2012 13:10:33 GMT

Hmm... I believe this script it a bit dangerous. What happens, if you have 3 events (one unique and two, which is duplicates) within the same second? Then the script will delete one of the duplicates AND the single unique event, right?