Splunk Search

Is there a way to chain searches?

daniel333
Builder

All,

A user just asked me this, any ideas on how to do this?

Splunkj Q: is the following supported?
I create an alert with a search trigger condition. If alert is triggered, run a different query, dump the results of that second query in an email to me
i.e., I want an alert to trigger a secondary search (because this additional search would contain useful diag. info on the main problem that triggered the alert originally) and have that secondary search data as the email (edited)


concretely
alert trigger: index=siebel "SBL-EAI-04117" earliest=-5m | stats count | where count>200 (edited)
if triggered, execute: index=siebel "SBL-EAI-04117" earliest=-60m | bin _time span=5m | stats count by _time | sort _time
(if error count threshold breached, show me the last 60 minutes of data in an email)

Tags (2)

pradeepkumarg
Influencer

You could do this in just one search

index=siebel "SBL-EAI-04117" earliest=-60m | bin _time span=5m | stats count by _time | sort _time | eventstats first(count) as last_5min | search last_5min > 200
0 Karma

elliotproebstel
Champion

Might be a better way, but this should work:

index=siebel "SBL-EAI-04117" earliest=-60m 
| bin _time span=5m 
| stats count by _time 
| sort _time
| eventstats latest(count) AS 5m_count
| where '5m_count'>200
| fields - '5m_count'

And then send an alert if the result count is not zero. The eventstats line will find the count from the latest 5 minutes, apply that count to all events, and then retain only the events where the 5m_count field is greater than 200. Because it'll be the same value for all events, it will either retain all events or throw away all of them.

pradeepkumarg
Influencer

Saw your answer after submitting mine. Pretty close 🙂

elliotproebstel
Champion

Gives me more confidence that this is the right way! 🙂 Great minds and all that...

Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...