topic hide the duplicate events in Splunk Search

hide the duplicate events

rbw78 — Fri, 06 Jan 2012 15:02:55 GMT

Hello,

I have 2 sources of events with "almost" the same framework and some of them reference the same event with the same content.
I want to hide in my search the events which matchs between the both sources for seeing only the single events.
I don't know what kind of function i have to use to do that (dedup, correlate ????)

Here is a single event from the 2 differents sources

Event in source 1

Jan  5 14:11:09 10.233.118.0 Jan  5 14:11:09 LX2AW8DSY CEF:0|Trend Micro|OfficeScan Intrusion Defense Firewall|6.1.0.69|1004371|Mozilla Firefox Obfuscated URLs Within Iframes Vulnerability|6|dmac=F0:DE:F1:18:58:49 smac=00:21:55:EF:8F:FF src=10.5.69.50 dst=10.233.118.0 in=1414 proto=TCP spt=8080 dpt=63034 cs2=0x00 ACK cnt=1 act=IDS:Log cn3=503 cs5=761296 cs1=Obfuscated_URL_i cs6=8

Event in source 2

1/5/2012 14:11,LX2AW8DSY,DPI Rule: 1004371 - Mozilla Firefox Obfuscated URLs Within Iframes Vulnerability,Web Client Mozilla FireFox,Detect Only: Log,25,Reverse Flow,F0:DE:F1:18:58:49,TCP,ACK,10.5.69.50,00:21:55:EF:8F:FF,8080,10.233.118.0,F0:DE:F1:18:58:49,63034,1400,"""Obfuscated URL i""",0,0,,1,

I know i can use the date, hours, minutes and type of event to check if there's a duplicate, doing a dedup of the 2 events and hide it.

How could i do that ?

thanks

Re: hide the duplicate events

Ayn — Fri, 06 Jan 2012 19:07:58 GMT

Use dedup with the fields you want to dedup on as arguments. So for instance if you have a field rule in both events and want to filter multiple events within the same day of the month, hour and minute, you would do:

... | dedup rule date_mday date_hour date_minute

EDIT: Sorry, I see now that I misunderstood your question, I didn't realize you wanted to remove BOTH events. For that I would advise you to use transaction in a similar way:

... | transaction rule maxspan=1m | search eventcount=1

This will remove all transactions which have more than one event (i.e. the ones with duplicate messages).

Re: hide the duplicate events

rtadams89 — Fri, 06 Jan 2012 20:58:13 GMT

Dedup is still going to return one of the events though. If you want to return only those events that are not in both sets, you could:

Perform a search taht returns both event types in the same result set. Add a field to each event that identifies what "type" or source each event comes from. Maybe | eval type1=if(index=1, "TRUE", "FALSE") | eval type2=if(index=2, "TRUE", "FALSE") |. Then use the transaction command to merge similar events into a single event. Finally, add | where NOT (type1=TRUE AND type2=TRUE) to the end of the search string.