Splunk Search

How to combine two searches to form an alert

alexrieffel
Observer

Query 1:

(sourcetype="PAYA:Enterprise:CDE:Web:App:Gateway.Bankcard" OR sourcetype="PAYA:Enterprise:CDE:Web:App:Gateway.ACH") AND Processing response: | stats count by host | eventstats sum(count) as totalTransactions | eval percent=round(count*100/totalTransactions,2) | eval transPerMinute=round(totalTransactions/10) | where percent>30 AND transPerMinute>200

Query 2:

(sourcetype="PAYA:Enterprise:CDE:Web:App:Gateway.Bankcard" OR sourcetype="PAYA:Enterprise:CDE:Web:App:Gateway.ACH") AND tsys_response_time>5000 | stats count by host

Basically, I need to create an alert that if one web server has processed over 30% of the transactions in the past 10 minutes, and we are averaging over 200 transactions per minute... AND if it has two or more transactions over 5000ms

I've been wrapping my brain around this for a long time... really hoping someone can help 🙂

0 Karma

to4kawa
SplunkTrust
SplunkTrust
(sourcetype="PAYA:Enterprise:CDE:Web:App:Gateway.Bankcard" OR sourcetype="PAYA:Enterprise:CDE:Web:App:Gateway.ACH") 
AND Processing response: AND tsys_response_time>5000
| stats count by host 
| eventstats sum(count) as totalTransactions 
| eval percent=round(count*100/totalTransactions,2) 
| eval transPerMinute=round(totalTransactions/10) 
| where percent>30 AND transPerMinute>200

Simply eventcount > 2, fire alert.
maybe some code disappear.

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

You'll have to correct the first case statement below to test for your first kind of record, but this is the general solution:

Index=foo 
(sourcetype="PAYA:Enterprise:CDE:Web:App:Gateway.Bankcard" OR sourcetype="PAYA:Enterprise:CDE:Web:App:Gateway.ACH") 
| eval CountProc = case( this/is/a/Processing response:/thing, 1)
| eval CountResp = case(tsys_response_time>5000,1)
| stats sum(CountProc) as CountProc 
        sum(CountResp) as CountResp 
   by host

| eventstats sum(CountProc) as totalTransactions 
| eval percent=round(CountProc*100/totalTransactions,2) 
| eval transPerMinute=round(totalTransactions/10) 
| where percent>30 AND transPerMinute>200 and CountResp>=2

Description in English of the above:

A) collect all the data you want from multiple kinds of records

B) calculate different fields that will be missing and/or zero when you don't want to count them, present and/or 1 when you do

C) run a single stats command that rolls them together

D) do your calculations and presentations

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

If you mark your code, with with the 101/010 button or by putting at least three spaces before each line or by putting three accents before and after it -

 ```code in here```

then the HTML monster will not swallow your HTML-like code.

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.