Splunk Search

How to create a panel that displays response events which take longer than the time of X to respond?

helpmejesus
Explorer

Hey fellow Splunkers,

I have a very complex problem which I am attempting to solve and thought it couldn't hurt to ask publicly since I received such awesome support the last time I had an issue.

I have analyzed a sample size of handshakes between log servers 1 and 2, and have found that the average time it takes server 2 to respond to Server 1 is approximately 1 second. The apex time of a normal handshake being approximately 1.6 seconds, I have decided that any handshake which takes longer than 2 seconds is an interesting event and would like to have a means of keeping track of them. To extrapolate further, a SYN/ACK event looks like the following:

SYN event:

2016/04/15 13:30:03.624 I-100001 TCPCOM1   F01234 😘   TDATA D(123,**100,,,,**12,Credit,1231231,12312,1,Sale,,,(UV)111111******1111(UV),1231,111111******1111=1231*************,123,1231,,,,*Tokenization BLBLAWORD *CEM_Words,,123123,,,,,,,,,,,,,,,,,,,,,,,,,,English,12312312,123123,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,BLAable,,) N(6,F01234 -> FIPFRED)

ACK Event:

2016/04/15 13:30:04.374 I-100001 TCPCOM1   F01234 😘   TSEND D(123,**101,,,**0,86,Credit,1231231,123123,1,Sale,,,(UV)111111******1111(UV),1212,111111******1111=1212*************,123,1212,,,,*Tokenization BLABLAHWords *CEM_Words *Wordtype=F D *Morewords=F  ,,123123,,,,,,,,,,1111111111111111,,,,123123,Transaction Approved ,,,,,,E123132132132132LZB7___,,123123,,123123,English,12312312,123123,,00,MPSCRD1,1,,12312312,,,,,,123,123123123,,,,,,,,,,,,,,,,MPSCRD1 123123 1231 123123 123123123123 00 001231,,Transaction Approved ,,,,0CE123,,,Wrdable,,) N(1,F01234)

Given the sensitive nature of my data, I have replaced all the numeric values and added figures to the end which shouldn't effect the end search. The important values which I care about are the date, the time, the site # (F01234), the 100/101,,, which distinguishes between SYN and ACK events (100 is SYN and 101 is ACK) and the value 111111******1111 and 1111111111111111 which I need to uniquely identify handshakes and tie two events together. I have annotated these values with (UV) for Unique Value, the (UV) will not be present in normal events.

So to summarize and clarify, I want to see all handshakes which take longer than 2 seconds, so I need to somehow use rex to grab events in pairs of two, separate them into evals 100 & 101, subtract 100 from 101 and push any events whose difference is greater than 2 into an eval which will represent alerts and ultimately populate my panel.

This is what I have thus far. I am still a Splunk newbie so this is an ugly mess and for that I apologize.

What I have that kind of works:

index=aaa source="D:\\aaa\\logs\\F*.trc.log" | rex "^(\S+\s+){2}(?<Time>\S+)\s+:" | rex "^(\S+\s+){4}(?<Store>\S+)\s+:" | rex "^(?:[^\*\n]*\*){7}(?P<**uniqueID**>\d+)" | eval Event=case(match(_raw,"100,,,"),"SYN",match(_raw,"101,,,"),"ACK") | transaction Store,uniqueID maxevents=2 maxspan=3s | table Store, duration, Event, _time

As far as I can tell this at least is pulling all the data I need. What I am having problems with is separating the events firstly by their unique ID then secondly by whether they are a 100 or 101. My current solution makes a lot of mistakes and misses like events. If anyone can help me re-mediate and refine my search it would be greatly appreciated.

0 Karma
1 Solution

somesoni2
SplunkTrust
SplunkTrust

How about this

index=aaa source="D:\\aaa\\logs\\F*.trc.log" | rex "^(\S+\s+){2}(?<Time>\S+)\s+:" | rex "^(\S+\s+){4}(?<Store>\S+)\s+:" | rex "^(?:[^\*\n]*\*){7}(?P<**uniqueID**>\d+)" | eval Event=case(match(_raw,"100,,,"),"SYN",match(_raw,"101,,,"),"ACK") | transaction Store,uniqueID startswith=Event="SYN" endswith=Event="ACK" maxevents=2 maxspan=3s | table Store, duration, Event, _time | where duration>2

View solution in original post

somesoni2
SplunkTrust
SplunkTrust

How about this

index=aaa source="D:\\aaa\\logs\\F*.trc.log" | rex "^(\S+\s+){2}(?<Time>\S+)\s+:" | rex "^(\S+\s+){4}(?<Store>\S+)\s+:" | rex "^(?:[^\*\n]*\*){7}(?P<**uniqueID**>\d+)" | eval Event=case(match(_raw,"100,,,"),"SYN",match(_raw,"101,,,"),"ACK") | transaction Store,uniqueID startswith=Event="SYN" endswith=Event="ACK" maxevents=2 maxspan=3s | table Store, duration, Event, _time | where duration>2

helpmejesus
Explorer

That helped, much appreciated

0 Karma

sundareshr
Legend

Try this

index=ajb source="D:\\AJBRTSV4\\logs\\F*.trc.log" | rex "^(\S+\s+){2}(?<Time>\S+)\s+:" | rex "^(\S+\s+){4}(?<Store>\S+)\s+:" | rex "^(?:[^*\n]*){7}(?P<uniqueID*>\d+)" | eval Event=case(match(_raw,"100,,,"),"SYN",match(_raw,"101,,,"),"ACK") | chart  list(Time) as Time over Store, uniqueid by Event | eval handshake=ACK-SYN | where handshake>120
0 Karma

helpmejesus
Explorer

So the UniqueID segment looks like its producing alot of false positives, I am trying to get it working but Its really hard to test. =(

0 Karma

helpmejesus
Explorer

Thanks for the help!

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...