"Duplicate" events: Without using "dedup", how can I remove all but one of the events that occur within a 1 second time frame?

[dbcase]

Hi,

I have events that are sorta kinda duplicated. Sorta kinda means that everything is the same EXCEPT there is a 1 second difference in the _time field. I can't use the dedup command because I would be eliminate other events that are interesting. How can I remove all but one of the events that occur within the 1 second?

[nzambo_splunk · Accepted solution]

Have you looked at transactions?
https://docs.splunk.com/Documentation/Splunk/6.5.1/SearchReference/Transaction

You could specify a maxspan of 1 or 2 seconds and key off of a particular field to group those events together. That would create a single event from multiple events. You may need to do some multi-value field voodoo to make sure the output of the fields is a single value but it should get you what you are looking for.

something like
index=mydata | transaction sourcetype maxspan=2s | eval myfield=mvdedup(mymvfield)

It's heavy depending on the volume of events, but might do what you need.

[dbcase]

This is what the final query looks like . Thanks for the help!!!

index=cox stuck OR unstuck  | rex "(GET|POST)\s(?<URL>\S+)"  | rex "(?<threadStatus>(STUCK|unstuck))"| rex "(?:.*?ExecuteThread:\s'){2}(?<threadID>\S+)[']"  | eval timestamp=strftime(_time,"%x %X")| sort _time| dedup threadID host _time | transaction URL threadStatus threadID maxspan=1m|sort _time|stats list(URL) as URL list(timestamp) as Time list(threadStatus) as "Thread Status" by host threadID|sort host threadID|fillnull value=NULL

[nzambo_splunk]

Thats's funny, I was just coming here to ask you how you ended up crafting the SPL since you marked this as an answer. Very glad to help! Just know that transactions can be very CPU intensive depending on the timerange and number of events, so use with caution.