How to search these events that meet the condition of "3 same contents(except time message) in 2 seconds", give me a idea, if possible, please.
our logs have timestamps in front. for example:
the original file:
"
Oct 31 2014 14:31:11 xiaowang ;Last diagnostic message repeated 14 times.
Oct 31 2014 14:34:11 xiaowang %%01INFO/4/SUPPRESS_DIAGLOG(D):CID=0x80600406;
Oct 31 2014 14:34:11:012 xiaowang %%01INFO/4/SUPPRESS_DIAGLOG(D):CID=0x80600406;
Oct 31 2014 14:34:12 xiaowang %%01INFO/4/SUPPRESS_DIAGLOG(D):CID=0x80600406;
Oct 31 2014 14:35:22 xiaowang %%01INFO/4/SUPPRESS_DIAGLOG(D):CID=0x80600406;
Oct 31 2014 14:34:32 xiaowang %%01RMBASE/6/PARTNER_FSM_TRANSFER(D):CID=80;
Oct 31 2014 14:34:56 xiaowang %%01DEBUG/4/DBG_TOP_PROC_MEM(D);
"
we hope to get these logs that have the same content(except the timestamps in front), but the time info in front may be different and the time interval is within 2 second. if we can find 3 logs that meet the frontal condition, we record it.
the expected result is:
"
Oct 31 2014 14:34:11 xiaowang %%01INFO/4/SUPPRESS_DIAGLOG(D):CID=0x80600406;
Oct 31 2014 14:34:11:012 xiaowang %%01INFO/4/SUPPRESS_DIAGLOG(D):CID=0x80600406;
Oct 31 2014 14:34:12 xiaowang %%01INFO/4/SUPPRESS_DIAGLOG(D):CID=0x80600406;
"
this question block me very long time, How, if at all possible, could this be accomplished?
attention: the time info format of logs may be different, for example: "Oct 31 2014 14:34:11" and "Oct 31 2014 14:34:11:012".
I want to 'transaction' and 'top' to deal with the problem, but I cannot devide a log item into two parts, one is timestamps in front and the other is log main content after the time info.
any advice?
Hi wangweibee,
the timestamp should not be a problem at all for Splunk, but you should get your content into a field first; try this:
your base search here | rex "\s\w+\s(?<myFoo>%.+)" | ...
This creates a new field called myFoo
(assuming all your interesting events start with a %
) which can be used in any further search command. So add a transaction
to it and your done:
your base search here | rex "\s\w+\s(?<myFoo>%.+)" | transaction "myFoo"
If this is what you want, use the regex and setup automatic field extraction http://docs.splunk.com/Documentation/Splunk/6.2.0/Knowledge/Createandmaintainsearch-timefieldextract... so you would not need the rex
in your search.
hope this helps ...
cheers, MuS
Hi wangweibee,
the timestamp should not be a problem at all for Splunk, but you should get your content into a field first; try this:
your base search here | rex "\s\w+\s(?<myFoo>%.+)" | ...
This creates a new field called myFoo
(assuming all your interesting events start with a %
) which can be used in any further search command. So add a transaction
to it and your done:
your base search here | rex "\s\w+\s(?<myFoo>%.+)" | transaction "myFoo"
If this is what you want, use the regex and setup automatic field extraction http://docs.splunk.com/Documentation/Splunk/6.2.0/Knowledge/Createandmaintainsearch-timefieldextract... so you would not need the rex
in your search.
hope this helps ...
cheers, MuS
thanks! "rex" can solve it.
but, now I need to search consecutive and same logs. that is, In addition to meet "3 same contents(except timestamps in front) in 2 seconds", the events must sequent in position, for example:
the original log:
"
Oct 31 2014 14:34:08 repeated 114 times.
Oct 31 2014 14:34:11 xiaowang CID=0x80600406; repeated 4 times.
Oct 31 2014 14:34:11 xiaowang CID=0x80600406; repeated 4 times.
Oct 31 2014 14:34:11 repeated 114 times.
Oct 31 2014 14:34:11 xiaowang CID=0x80600406; repeated 4 times.
Oct 31 2014 14:34:15 xiaowang CID=0x80600406; repeated 4 times.
Oct 31 2014 14:34:22 xiaowang CID=0x80600406; repeated 4 times.
Oct 31 2014 14:34:24 xiaowang CID=0x80600406; repeated 4 times.
"
the hindermost 4 events meet the criteria, and we get the result:
"
Oct 31 2014 14:34:11 xiaowang CID=0x80600406; repeated 4 times.
Oct 31 2014 14:34:15 xiaowang CID=0x80600406; repeated 4 times.
Oct 31 2014 14:34:22 xiaowang CID=0x80600406; repeated 4 times.
Oct 31 2014 14:34:24 xiaowang CID=0x80600406; repeated 4 times.
"
how to realize it ? thanks for any advice!
this problem is urgent, please give me some help.
I want to create a new posting, but it I have not enough reputation.
thanks.
I need not timestamps that will is weeded. I need the content after the timestamps.
the regex of "rex field=_raw "\w{3} \d{2} \d{4} \d{2}:\d{2}:\d{2} (?.)" " can match a kind of timestamps format, but cannot match other time format.
And the regex of "rex field=_raw "_time (?.)"" does not work。
how to utilize the splunk timestamps field distilled method in rex expression ?
based on your provided examples use this to match everything after your events timestamp:
rex "\:\d{2,3}\s(?<myField>.*)"
thanks for all!
but, how to use regex to express the timestamps that have many format, such as "Oct 31 2014 14:34:11", '14/10/31 14:35:22.000' and "Oct 31 2014 14:34:11:012", and the content after timestamps have arbitrary format, and there are not delimiter between the timestamps and the hinder log content.
thanks!
no need to regex for timestamps, use _time
as field. If your events have different formats and/or no delimiter you have to use these events in any online regex tester tool like www.regexr.com to get the matching regex.