How to identify all the duplicated log events/entries written to two different log files?

[myli12]

I have two types of log files, one is supposed to record security related events (sourcetype = sec) and the other to record the rest system events (sourcetype = sys).

But I found a small set of security events cross written to both types of the files. One of such log example is as follows (I displayed host and sourcetype using splunk)

2011-06-06T18:16:29 127.0.0.1 [NET-SECURITY]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/service radiusd restart
host=TT_June2011 Options| sourcetype = sec Options

2011-06-06T18:16:29 127.0.0.1 [NET-SECURITY]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/service radiusd restart
host=TT_June2011 Options| sourcetype = sys Options

How to identify all the log events/entries written to both of the files. At least is it possible to find if the security events written to the "sys" type file is a subset of all the security events written to the "sec" type file?

The suggested "punct" does not work. do I need to write python script to compare and identify the duplicate messages written to both files?

[myli12 · Accepted solution]

I found a solution throught this post.

http://splunk-base.splunk.com/answers/432/how-do-i-find-all-duplicate-events

The search that works is

sourcetype=sec OR sourcetype=sys | eval dupfield=_raw | transaction dupfield maxspan=1s keepevicted=true | where mvcount(sourcetype) > 1

[MarioM]

what about splunk special field punct ?

You could try a search similar to this :

sourcetype="sec" OR sourcetype="sys" | where mvcount(punct) > 1 | where mvcount(sourcetype) > 1

[myli12]

thanks. why did I receive "0" matching event after running the above suggested search. I obtained "0" matching event even after the first where, i.e. after where mvcount(punct) > 1? However, I can observe the idential log entries in the both sourcetypes (they are from differnet files).