Are you a member of the Splunk Community?
- Find Answers
- :
- Using Splunk
- :
- Splunk Search
- :
- is it possible to run remove duplicates on specifi...
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
When using the remove duplicate event python is it possible to run it on specific date range?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I create new script for you. http://pastebin.com/JusgWRMy
Usage: ./splunk cmd python rem_dupl_event_spec_date.py <index> <earliest_date> <latest_date>
earliest_date & latest_date syntax: %m/%d/%Y:%H:%M:%S
For example:
./splunk cmd python rem_dupl_event_spec_date.py test 05/20/2011:0:0:0 05/24/2011:23:59:59
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I create new script for you. http://pastebin.com/JusgWRMy
Usage: ./splunk cmd python rem_dupl_event_spec_date.py <index> <earliest_date> <latest_date>
earliest_date & latest_date syntax: %m/%d/%Y:%H:%M:%S
For example:
./splunk cmd python rem_dupl_event_spec_date.py test 05/20/2011:0:0:0 05/24/2011:23:59:59
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hmm... I believe this script it a bit dangerous. What happens, if you have 3 events (one unique and two, which is duplicates) within the same second? Then the script will delete one of the duplicates AND the single unique event, right?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Really like this. Is there a way for it to only look at a source or sourcetype within a given index?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
works great , Thanks!
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Found it..
Edited the script and added to line 56 (starting with search1) earliest= latest dates :
"search1 = 'search index=' + index + 'earliest=129461760 latest=1294704000 | ..."
I don't know Perl , anyone knows how to add these dates as parameters ?
.conf25 Global Broadcast: Don’t Miss a Moment
Observe and Secure All Apps with Splunk
What's New in Splunk Observability - August 2025
