Splunk Search
Highlighted

How to create a panel that displays response events which take longer than the time of X to respond?

Explorer

Hey fellow Splunkers,

I have a very complex problem which I am attempting to solve and thought it couldn't hurt to ask publicly since I received such awesome support the last time I had an issue.

I have analyzed a sample size of handshakes between log servers 1 and 2, and have found that the average time it takes server 2 to respond to Server 1 is approximately 1 second. The apex time of a normal handshake being approximately 1.6 seconds, I have decided that any handshake which takes longer than 2 seconds is an interesting event and would like to have a means of keeping track of them. To extrapolate further, a SYN/ACK event looks like the following:

SYN event:

2016/04/15 13:30:03.624 I-100001 TCPCOM1   F01234 😘   TDATA D(123,**100,,,,**12,Credit,1231231,12312,1,Sale,,,(UV)111111******1111(UV),1231,111111******1111=1231*************,123,1231,,,,*Tokenization BLBLAWORD *CEM_Words,,123123,,,,,,,,,,,,,,,,,,,,,,,,,,English,12312312,123123,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,BLAable,,) N(6,F01234 -> FIPFRED)

ACK Event:

2016/04/15 13:30:04.374 I-100001 TCPCOM1   F01234 😘   TSEND D(123,**101,,,**0,86,Credit,1231231,123123,1,Sale,,,(UV)111111******1111(UV),1212,111111******1111=1212*************,123,1212,,,,*Tokenization BLABLAHWords *CEM_Words *Wordtype=F D *Morewords=F  ,,123123,,,,,,,,,,1111111111111111,,,,123123,Transaction Approved ,,,,,,E123132132132132LZB7___,,123123,,123123,English,12312312,123123,,00,MPSCRD1,1,,12312312,,,,,,123,123123123,,,,,,,,,,,,,,,,MPSCRD1 123123 1231 123123 123123123123 00 001231,,Transaction Approved ,,,,0CE123,,,Wrdable,,) N(1,F01234)

Given the sensitive nature of my data, I have replaced all the numeric values and added figures to the end which shouldn't effect the end search. The important values which I care about are the date, the time, the site # (F01234), the 100/101,,, which distinguishes between SYN and ACK events (100 is SYN and 101 is ACK) and the value 111111******1111 and 1111111111111111 which I need to uniquely identify handshakes and tie two events together. I have annotated these values with (UV) for Unique Value, the (UV) will not be present in normal events.

So to summarize and clarify, I want to see all handshakes which take longer than 2 seconds, so I need to somehow use rex to grab events in pairs of two, separate them into evals 100 & 101, subtract 100 from 101 and push any events whose difference is greater than 2 into an eval which will represent alerts and ultimately populate my panel.

This is what I have thus far. I am still a Splunk newbie so this is an ugly mess and for that I apologize.

What I have that kind of works:

index=aaa source="D:\\aaa\\logs\\F*.trc.log" | rex "^(\S+\s+){2}(?<Time>\S+)\s+:" | rex "^(\S+\s+){4}(?<Store>\S+)\s+:" | rex "^(?:[^\*\n]*\*){7}(?P<**uniqueID**>\d+)" | eval Event=case(match(_raw,"100,,,"),"SYN",match(_raw,"101,,,"),"ACK") | transaction Store,uniqueID maxevents=2 maxspan=3s | table Store, duration, Event, _time

As far as I can tell this at least is pulling all the data I need. What I am having problems with is separating the events firstly by their unique ID then secondly by whether they are a 100 or 101. My current solution makes a lot of mistakes and misses like events. If anyone can help me re-mediate and refine my search it would be greatly appreciated.

0 Karma
Highlighted

Re: How to create a panel that displays response events which take longer than the time of X to respond?

Legend

Try this

index=ajb source="D:\\AJBRTSV4\\logs\\F*.trc.log" | rex "^(\S+\s+){2}(?<Time>\S+)\s+:" | rex "^(\S+\s+){4}(?<Store>\S+)\s+:" | rex "^(?:[^*\n]*){7}(?P<uniqueID*>\d+)" | eval Event=case(match(_raw,"100,,,"),"SYN",match(_raw,"101,,,"),"ACK") | chart  list(Time) as Time over Store, uniqueid by Event | eval handshake=ACK-SYN | where handshake>120
0 Karma
Highlighted

Re: How to create a panel that displays response events which take longer than the time of X to respond?

Explorer

So the UniqueID segment looks like its producing alot of false positives, I am trying to get it working but Its really hard to test. =(

0 Karma
Highlighted

Re: How to create a panel that displays response events which take longer than the time of X to respond?

Explorer

Thanks for the help!

0 Karma
Highlighted

Re: How to create a panel that displays response events which take longer than the time of X to respond?

SplunkTrust
SplunkTrust

How about this

index=aaa source="D:\\aaa\\logs\\F*.trc.log" | rex "^(\S+\s+){2}(?<Time>\S+)\s+:" | rex "^(\S+\s+){4}(?<Store>\S+)\s+:" | rex "^(?:[^\*\n]*\*){7}(?P<**uniqueID**>\d+)" | eval Event=case(match(_raw,"100,,,"),"SYN",match(_raw,"101,,,"),"ACK") | transaction Store,uniqueID startswith=Event="SYN" endswith=Event="ACK" maxevents=2 maxspan=3s | table Store, duration, Event, _time | where duration>2

View solution in original post

Highlighted

Re: How to create a panel that displays response events which take longer than the time of X to respond?

Explorer

That helped, much appreciated

0 Karma