Splunk Search

How to combine two searches to form an alert

alexrieffel
Observer

Query 1:

(sourcetype="PAYA:Enterprise:CDE:Web:App:Gateway.Bankcard" OR sourcetype="PAYA:Enterprise:CDE:Web:App:Gateway.ACH") AND Processing response: | stats count by host | eventstats sum(count) as totalTransactions | eval percent=round(count*100/totalTransactions,2) | eval transPerMinute=round(totalTransactions/10) | where percent>30 AND transPerMinute>200

Query 2:

(sourcetype="PAYA:Enterprise:CDE:Web:App:Gateway.Bankcard" OR sourcetype="PAYA:Enterprise:CDE:Web:App:Gateway.ACH") AND tsys_response_time>5000 | stats count by host

Basically, I need to create an alert that if one web server has processed over 30% of the transactions in the past 10 minutes, and we are averaging over 200 transactions per minute... AND if it has two or more transactions over 5000ms

I've been wrapping my brain around this for a long time... really hoping someone can help 🙂

0 Karma

to4kawa
Ultra Champion
(sourcetype="PAYA:Enterprise:CDE:Web:App:Gateway.Bankcard" OR sourcetype="PAYA:Enterprise:CDE:Web:App:Gateway.ACH") 
AND Processing response: AND tsys_response_time>5000
| stats count by host 
| eventstats sum(count) as totalTransactions 
| eval percent=round(count*100/totalTransactions,2) 
| eval transPerMinute=round(totalTransactions/10) 
| where percent>30 AND transPerMinute>200

Simply eventcount > 2, fire alert.
maybe some code disappear.

0 Karma

DalJeanis
Legend

You'll have to correct the first case statement below to test for your first kind of record, but this is the general solution:

Index=foo 
(sourcetype="PAYA:Enterprise:CDE:Web:App:Gateway.Bankcard" OR sourcetype="PAYA:Enterprise:CDE:Web:App:Gateway.ACH") 
| eval CountProc = case( this/is/a/Processing response:/thing, 1)
| eval CountResp = case(tsys_response_time>5000,1)
| stats sum(CountProc) as CountProc 
        sum(CountResp) as CountResp 
   by host

| eventstats sum(CountProc) as totalTransactions 
| eval percent=round(CountProc*100/totalTransactions,2) 
| eval transPerMinute=round(totalTransactions/10) 
| where percent>30 AND transPerMinute>200 and CountResp>=2

Description in English of the above:

A) collect all the data you want from multiple kinds of records

B) calculate different fields that will be missing and/or zero when you don't want to count them, present and/or 1 when you do

C) run a single stats command that rolls them together

D) do your calculations and presentations

0 Karma

DalJeanis
Legend

If you mark your code, with with the 101/010 button or by putting at least three spaces before each line or by putting three accents before and after it -

 ```code in here```

then the HTML monster will not swallow your HTML-like code.

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...