Splunk Search

How to use subsearch to only return server events that are NOT followed by a recovery event within a certain period of time?

lennys26
Communicator

I have a search that returns server events and would like to know when this event is NOT followed by a recovery message within a short period of time.

In my example below, the event is triggered and it recovers 5 seconds later. In this case, I would NOT want this to return results.

Oct 21 06:40:13 cam-vm-mon3 mfsmount[3425]: master: connection lost (1)
Oct 21 06:40:18 cam-vm-mon3 mfsmount[3425]: registered to master

To give my question better context, I am actually running this in an old Splunk 4.3 install and am configuring this as an alert, so these are alert criteria more than search criteria

I have been playing with append and appendpipes but am not having any luck. Can anyone offer any suggestions?

0 Karma
1 Solution

woodcock
Esteemed Legend

Like this (with a base search that limits to only 2 types of events):

... | eval type=case(searchmatch("master: connection lost"), "down", searchmatch("registered to master"), "up", true(), "BUG!") | reverse | streamstats count(eval(type="down")) AS sessionID by host | eventstats latest(_time) AS latestTime latest(type) AS latestType by sessionID  host | where type="down" | eval downSeconds=if(latestType="down", now(), latestTime) - _time | where downSeconds > 5

View solution in original post

0 Karma

woodcock
Esteemed Legend

Like this (with a base search that limits to only 2 types of events):

... | eval type=case(searchmatch("master: connection lost"), "down", searchmatch("registered to master"), "up", true(), "BUG!") | reverse | streamstats count(eval(type="down")) AS sessionID by host | eventstats latest(_time) AS latestTime latest(type) AS latestType by sessionID  host | where type="down" | eval downSeconds=if(latestType="down", now(), latestTime) - _time | where downSeconds > 5
0 Karma

lennys26
Communicator

Additional thoughts on paths forward.

  • Search for "master". Eval/search for "registered". Where this returns NULL, return details from Master search (or return something!), else, return NULL.

OR

  • In the past I have played (unsuccessfully) with the idea of simply using a counter to....say.... run a search for "master" and a sub-search for "registered". For each event matching "master" add 1 (to x), for each "registered" subtract 1. Then where x>0, do something
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...