Splunk Search

How to combine two searches to form an alert

alexrieffel
Observer

Query 1:

(sourcetype="PAYA:Enterprise:CDE:Web:App:Gateway.Bankcard" OR sourcetype="PAYA:Enterprise:CDE:Web:App:Gateway.ACH") AND Processing response: | stats count by host | eventstats sum(count) as totalTransactions | eval percent=round(count*100/totalTransactions,2) | eval transPerMinute=round(totalTransactions/10) | where percent>30 AND transPerMinute>200

Query 2:

(sourcetype="PAYA:Enterprise:CDE:Web:App:Gateway.Bankcard" OR sourcetype="PAYA:Enterprise:CDE:Web:App:Gateway.ACH") AND tsys_response_time>5000 | stats count by host

Basically, I need to create an alert that if one web server has processed over 30% of the transactions in the past 10 minutes, and we are averaging over 200 transactions per minute... AND if it has two or more transactions over 5000ms

I've been wrapping my brain around this for a long time... really hoping someone can help 🙂

0 Karma

to4kawa
Ultra Champion
(sourcetype="PAYA:Enterprise:CDE:Web:App:Gateway.Bankcard" OR sourcetype="PAYA:Enterprise:CDE:Web:App:Gateway.ACH") 
AND Processing response: AND tsys_response_time>5000
| stats count by host 
| eventstats sum(count) as totalTransactions 
| eval percent=round(count*100/totalTransactions,2) 
| eval transPerMinute=round(totalTransactions/10) 
| where percent>30 AND transPerMinute>200

Simply eventcount > 2, fire alert.
maybe some code disappear.

0 Karma

DalJeanis
Legend

You'll have to correct the first case statement below to test for your first kind of record, but this is the general solution:

Index=foo 
(sourcetype="PAYA:Enterprise:CDE:Web:App:Gateway.Bankcard" OR sourcetype="PAYA:Enterprise:CDE:Web:App:Gateway.ACH") 
| eval CountProc = case( this/is/a/Processing response:/thing, 1)
| eval CountResp = case(tsys_response_time>5000,1)
| stats sum(CountProc) as CountProc 
        sum(CountResp) as CountResp 
   by host

| eventstats sum(CountProc) as totalTransactions 
| eval percent=round(CountProc*100/totalTransactions,2) 
| eval transPerMinute=round(totalTransactions/10) 
| where percent>30 AND transPerMinute>200 and CountResp>=2

Description in English of the above:

A) collect all the data you want from multiple kinds of records

B) calculate different fields that will be missing and/or zero when you don't want to count them, present and/or 1 when you do

C) run a single stats command that rolls them together

D) do your calculations and presentations

0 Karma

DalJeanis
Legend

If you mark your code, with with the 101/010 button or by putting at least three spaces before each line or by putting three accents before and after it -

 ```code in here```

then the HTML monster will not swallow your HTML-like code.

0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...