Splunk Search

What should be my condition that will trigger the alert action, for a search string below

Snehaan
Explorer

Hello,

I have a search string like below, where it is fetching data from stage and giving out aggregates of Trades for each source system in that stage. 

index=qrp STAGE IN (TRADE_EVENT)
| bucket _time span=1h
| timechart useother=f span=1h sum(TRADES) as "TradeCount" by ODS_SRC_SYSTEM_CODE
| fillnull value=0

The results are fetched like below:

TradeCount:A1   TradeCountA2   TradeCountA3

27                                 5                                 0

What should be my condition that will trigger the alert action, when sum(trades) for a source system is 0? Example: TradeCountA3 is having value Zero now. 

Labels (2)
0 Karma
1 Solution

rnowitzki
Builder

Ah ok, that's what I assumed.  So there is no prefix that catches them all.

So my last SPL, with the "foreach *" should work for you.

BR
Ralph
--
Karma and/or Solution tagging appreciated.

 

--
Karma and/or Solution tagging appreciated.

View solution in original post

0 Karma

rnowitzki
Builder

Hi @Snehaan ,

Do you know the possible values for ODS_SRC_SYSTEM_CODE? Is it dynamic, or a pretty set range of values?

If you know them, and it's not like hundreds, you could add a custom trigger and put it like.

TradeCountA1=0 OR TradeCountA2=0 OR TradeCountA3=0 ...


Another option is to add a Column that gets the smallest value of all TradeCount* fields for each row.
If this new field=0 you can trigger the alert, because at least one of them has "0".

| foreach TradeCount*
    [eval Min=case(Min<<<FIELD>>,Min,true(),<<FIELD>>)]

 
Hope it helps.
BR
Ralph

--
Karma and/or Solution tagging appreciated.

--
Karma and/or Solution tagging appreciated.
0 Karma

Snehaan
Explorer

Hi @rnowitzki  Thank you for your reply. The 2nd solution is a great idea! I wanted to know is this the way my search string should look like now after appending your code?

index=qrp STAGE IN (TRADE_EVENT)
| bucket _time span=1h
| timechart useother=f span=1h sum(TRADES) as "TradeCount" by ODS_SRC_SYSTEM_CODE
| fillnull value=0 | foreach TradeCount* [eval Min=case(Min<<<FIELD>>,Min,true(),<<FIELD>>)]

When I search this I am not able to see any new column. How should it look like. I am very new to Splunk and trying to figure out things. Kindly help. 🙂 

0 Karma

rnowitzki
Builder

Can you paste how your table looks like after your timechart command. I guess you don't see a new column, because the field/columns are not titled with TradeCountsomething.  Is the title maybe just A1, A2, A3?

Should also work with just "*", so like:

index=qrp STAGE IN (TRADE_EVENT)
| bucket _time span=1h
| timechart useother=f span=1h sum(TRADES) as "TradeCount" by ODS_SRC_SYSTEM_CODE
| fillnull value=0 
| foreach * [eval Min=case(Min<<<FIELD>>,Min,true(),<<FIELD>>)]

 
If not, please copy the result table as you see it in Splunk. Maybe even as Screenshot.

BR
Ralph
--
Karma and/or Solution tagging appreciated.

--
Karma and/or Solution tagging appreciated.
0 Karma

Snehaan
Explorer

Snehaan_0-1596795344474.png

 

0 Karma

Snehaan
Explorer

@rnowitzki  originally it looks like this.

0 Karma

Snehaan
Explorer

Hi @rnowitzki 

Your search string with Foreach * is working. I could see the last column Min which holds the minimum value for the entire row.  

So now for alert scheduling the below deatils should work right?

  • Trigger condition: Custom
  • Custom Condition: Min =0 
0 Karma

rnowitzki
Builder

Hi @Snehaan ,

yes, exactly. That should do it.

BR
Ralph
--
Karma and/or Solution tagging appreciated.

--
Karma and/or Solution tagging appreciated.
0 Karma

Snehaan
Explorer

@rnowitzki 

Many thanks for your help. 🙂

Could you also please suggest a good document to read and understand how to write search string based on the data stored behind, like in my example.

0 Karma

rnowitzki
Builder

Glad I could help.
You could mark my answer with the "foreach *" as Solution, so that it's visible for others that it was solved.

I would recommend doing at least the (free) Fundamentals I training as a starter.
https://www.splunk.com/en_us/training/courses/splunk-fundamentals-1.html

Cheers
Ralph
--
Karma and/or Solution tagging appreciated.

--
Karma and/or Solution tagging appreciated.
0 Karma

Snehaan
Explorer
Spoiler
Hi @rnowitzki  thank you for sharing the document link.

I also had one more query related to same question. Earlier we were searching wherever value is 0 for at least one source system send alert. But what if there is a minimum threshold value for each source system, and if we want to set alert whenever any one out of 5 source system has trade values below threshold limit.
Note: different source system have different thresholds.
0 Karma

rnowitzki
Builder

Hi @Snehaan ,

You could probably solve it with an eval/if statement, that checks all the values.

eval alert = if(system1_value >= 2 OR system2_value >= 5 OR system3_value >= 6 ..., 1 , 0)

 

So, when any of the checks is true, that "alert" field will have "1" as value, if none of the systems reached its threshold, it will have 0.  You can then base the trigger for the alert on this "alert" field. 

This is not dynamic, meaning you will have to know which systems are expected and set the thresholds accordingly.

Hope it helps.
BR
Ralph

--
Karma and/or Solution tagging appreciated.
0 Karma

Snehaan
Explorer

Hi @rnowitzki 

Yes, I  know the list of source systems for my use case, and the below solution works perfect for my case. 

Thanks a lot! 🙂

 

0 Karma

Snehaan
Explorer

Hi @rnowitzki , I was thinking of one more use case as below:

If at 8 am the search string gives data like below:

A1         A2          A3

100      200        550

At 9 am the search string gives data like below:

A1       A2          A3

10      400       350

Then from 8am to 9 am, there is drop in the values for A1 and A3.

Can we set an alert by comparing between two time frames and if value in previous time frame is more than the next one then schedule alerts?

0 Karma

rnowitzki
Builder

Hi @Snehaan ,

There are several ways to achieve this, search for compare time frames  etc, in the community  to get some inspiration.

This is one way to do it:

Put the search time to from:-2h@h to  @h so that you have 2 full hours to compare.

|timechart span=1h count by systemcode
| delta A1
| delta A2
| delta A3
| fields delta*

 

Now you have the difference between the last and current hour. 

Now you could use the "foreach" logic again to find the minimum value and fire the alarm if there is anything <0.

It's a not very dynamic solution. There will be better ones, but should work with your systems (A1-A3) are not changing.

 

BR 
Ralph

--
Karma and/or Solution tagging appreciated.
0 Karma

Snehaan
Explorer

Hi @rnowitzki  thank you for the solution. I will try this out. 🙂

0 Karma

rnowitzki
Builder

Ah ok, that's what I assumed.  So there is no prefix that catches them all.

So my last SPL, with the "foreach *" should work for you.

BR
Ralph
--
Karma and/or Solution tagging appreciated.

 

--
Karma and/or Solution tagging appreciated.
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...