If two events occur X seconds apart ignore the first one

[dbcase]

Hi,

I have a log file that reports an event twice. It is the exact same event except it is repeated 1 or 2 or 3 or up to 5 seconds apart. It is only repeated twice. What I'd like to do is ignore the first event and report the second event.

[niketn]

Can you add your field names and some sample data?

[dbcase]

Sure!

Here is the query that I have

index=cox stuck OR unstuck  | rex "GET\s(?<URL>\S+)"  | rex "(?<threadStatus>(STUCK|unstuck))"| rex "(?:.*?ExecuteThread:\s'){2}(?<threadID>\S+)[']"  | eval timestamp=strftime(_time,"%x %X")| sort _time| dedup threadID host _time| stats list(URL) as URL list(timestamp) as Time list(threadStatus) as "Thread Status" by host threadID|sort host threadID

It generates a report that looks like this

host .               threadID           URL                                                                Time                     Thread Status

host_portal1    7                  /rest/icontrol/sites/72178/rules                01/24/17 03:01:03            STUCK
                                                  /rest/icontrol/sites/72178/rules                  01/24/17 03:02:03           STUCK
                                                                                                                            01/24/17 03:02:10           unstuck

If you notice the first two lines they are identical except for the 1 second differential in time. I'd like to eliminate one of the two lines (doesn't matter which one) so the report looks like this

host . threadID URL Time Thread Status

host_portal1    7                  /rest/icontrol/sites/72178/rules                01/24/17 03:01:03            STUCK
                                                                                                                           01/24/17 03:02:10           unstuck

Does that help?

[dbcase]

Ugh well the formatting kinda sucks but hopefully you can get the idea.....

[hunters_splunk]

Hi dbcase,

I think you can use the dedup command to remove deplicate events that contain identical combination of values for the fields that you specify. You can specify the number of events with duplicate values, or value combinations, to keep. You can sort the fields. When you sort, the dedup command deduplicates the results based on the specified sort-by fields.
For example, assuming you use clientip and action to identify events, you can use the following search:

... | dedup clientip action sortby +_time

For detailed information about the dedup command, please refer to documentation:
http://docs.splunk.com/Documentation/Splunk/6.5.1/SearchReference/Dedup
Hope this helps. Thanks!
Hunter

[dbcase]

Hi Hunters,

I thought about that but the challenge is the event is identical except for the time. So if I dedup and exclude the time I remove other events that I'm interested in. If I dedup and include the time it doesn't do anything because the time is unique.