Hey fellow Splunkers,
I have a very complex problem which I am attempting to solve and thought it couldn't hurt to ask publicly since I received such awesome support the last time I had an issue.
I have analyzed a sample size of handshakes between log servers 1 and 2, and have found that the average time it takes server 2 to respond to Server 1 is approximately 1 second. The apex time of a normal handshake being approximately 1.6 seconds, I have decided that any handshake which takes longer than 2 seconds is an interesting event and would like to have a means of keeping track of them. To extrapolate further, a SYN/ACK event looks like the following:
SYN event:
2016/04/15 13:30:03.624 I-100001 TCPCOM1 F01234 😘 TDATA D(123,**100,,,,**12,Credit,1231231,12312,1,Sale,,,(UV)111111******1111(UV),1231,111111******1111=1231*************,123,1231,,,,*Tokenization BLBLAWORD *CEM_Words,,123123,,,,,,,,,,,,,,,,,,,,,,,,,,English,12312312,123123,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,BLAable,,) N(6,F01234 -> FIPFRED)
ACK Event:
2016/04/15 13:30:04.374 I-100001 TCPCOM1 F01234 😘 TSEND D(123,**101,,,**0,86,Credit,1231231,123123,1,Sale,,,(UV)111111******1111(UV),1212,111111******1111=1212*************,123,1212,,,,*Tokenization BLABLAHWords *CEM_Words *Wordtype=F D *Morewords=F ,,123123,,,,,,,,,,1111111111111111,,,,123123,Transaction Approved ,,,,,,E123132132132132LZB7___,,123123,,123123,English,12312312,123123,,00,MPSCRD1,1,,12312312,,,,,,123,123123123,,,,,,,,,,,,,,,,MPSCRD1 123123 1231 123123 123123123123 00 001231,,Transaction Approved ,,,,0CE123,,,Wrdable,,) N(1,F01234)
Given the sensitive nature of my data, I have replaced all the numeric values and added figures to the end which shouldn't effect the end search. The important values which I care about are the date, the time, the site # (F01234), the 100/101,,, which distinguishes between SYN and ACK events (100 is SYN and 101 is ACK) and the value 111111******1111
and 1111111111111111
which I need to uniquely identify handshakes and tie two events together. I have annotated these values with (UV) for Unique Value, the (UV) will not be present in normal events.
So to summarize and clarify, I want to see all handshakes which take longer than 2 seconds, so I need to somehow use rex to grab events in pairs of two, separate them into evals 100 & 101, subtract 100 from 101 and push any events whose difference is greater than 2 into an eval which will represent alerts and ultimately populate my panel.
This is what I have thus far. I am still a Splunk newbie so this is an ugly mess and for that I apologize.
What I have that kind of works:
index=aaa source="D:\\aaa\\logs\\F*.trc.log" | rex "^(\S+\s+){2}(?<Time>\S+)\s+:" | rex "^(\S+\s+){4}(?<Store>\S+)\s+:" | rex "^(?:[^\*\n]*\*){7}(?P<**uniqueID**>\d+)" | eval Event=case(match(_raw,"100,,,"),"SYN",match(_raw,"101,,,"),"ACK") | transaction Store,uniqueID maxevents=2 maxspan=3s | table Store, duration, Event, _time
As far as I can tell this at least is pulling all the data I need. What I am having problems with is separating the events firstly by their unique ID then secondly by whether they are a 100 or 101. My current solution makes a lot of mistakes and misses like events. If anyone can help me re-mediate and refine my search it would be greatly appreciated.
How about this
index=aaa source="D:\\aaa\\logs\\F*.trc.log" | rex "^(\S+\s+){2}(?<Time>\S+)\s+:" | rex "^(\S+\s+){4}(?<Store>\S+)\s+:" | rex "^(?:[^\*\n]*\*){7}(?P<**uniqueID**>\d+)" | eval Event=case(match(_raw,"100,,,"),"SYN",match(_raw,"101,,,"),"ACK") | transaction Store,uniqueID startswith=Event="SYN" endswith=Event="ACK" maxevents=2 maxspan=3s | table Store, duration, Event, _time | where duration>2
How about this
index=aaa source="D:\\aaa\\logs\\F*.trc.log" | rex "^(\S+\s+){2}(?<Time>\S+)\s+:" | rex "^(\S+\s+){4}(?<Store>\S+)\s+:" | rex "^(?:[^\*\n]*\*){7}(?P<**uniqueID**>\d+)" | eval Event=case(match(_raw,"100,,,"),"SYN",match(_raw,"101,,,"),"ACK") | transaction Store,uniqueID startswith=Event="SYN" endswith=Event="ACK" maxevents=2 maxspan=3s | table Store, duration, Event, _time | where duration>2
That helped, much appreciated
Try this
index=ajb source="D:\\AJBRTSV4\\logs\\F*.trc.log" | rex "^(\S+\s+){2}(?<Time>\S+)\s+:" | rex "^(\S+\s+){4}(?<Store>\S+)\s+:" | rex "^(?:[^*\n]*){7}(?P<uniqueID*>\d+)" | eval Event=case(match(_raw,"100,,,"),"SYN",match(_raw,"101,,,"),"ACK") | chart list(Time) as Time over Store, uniqueid by Event | eval handshake=ACK-SYN | where handshake>120
So the UniqueID segment looks like its producing alot of false positives, I am trying to get it working but Its really hard to test. =(
Thanks for the help!