Archive

How to identify all the duplicated log events/entries written to two different log files?

myli12
Path Finder

I have two types of log files, one is supposed to record security related events (sourcetype = sec) and the other to record the rest system events (sourcetype = sys).

But I found a small set of security events cross written to both types of the files. One of such log example is as follows (I displayed host and sourcetype using splunk)

2011-06-06T18:16:29 127.0.0.1 [NET-SECURITY]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/service radiusd restart
host=TT_June2011 Options| sourcetype = sec Options

2011-06-06T18:16:29 127.0.0.1 [NET-SECURITY]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/service radiusd restart
host=TT_June2011 Options| sourcetype = sys Options

How to identify all the log events/entries written to both of the files. At least is it possible to find if the security events written to the "sys" type file is a subset of all the security events written to the "sec" type file?

The suggested "punct" does not work. do I need to write python script to compare and identify the duplicate messages written to both files?

Tags (1)
1 Solution

myli12
Path Finder

I found a solution throught this post.

http://splunk-base.splunk.com/answers/432/how-do-i-find-all-duplicate-events

The search that works is

sourcetype=sec OR sourcetype=sys | eval dupfield=_raw | transaction dupfield maxspan=1s keepevicted=true | where mvcount(sourcetype) > 1

View solution in original post

myli12
Path Finder

I found a solution throught this post.

http://splunk-base.splunk.com/answers/432/how-do-i-find-all-duplicate-events

The search that works is

sourcetype=sec OR sourcetype=sys | eval dupfield=_raw | transaction dupfield maxspan=1s keepevicted=true | where mvcount(sourcetype) > 1

View solution in original post

MarioM
Motivator

what about splunk special field punct ?

You could try a search similar to this :

sourcetype="sec" OR sourcetype="sys" | where mvcount(punct) > 1 | where mvcount(sourcetype) > 1
0 Karma

myli12
Path Finder

thanks. why did I receive "0" matching event after running the above suggested search. I obtained "0" matching event even after the first where, i.e. after where mvcount(punct) > 1? However, I can observe the idential log entries in the both sourcetypes (they are from differnet files).

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!