Security

Search backwards matching on value in current search result

johnca00
New Member

Hello -

I have a logging event like this one. We are searching on "Threshold Exceeded" AND "225"

9/26/17 13:45:18:690 EDT] 000215d9 SystemOut     O 4580330012 [SIBJMSRAThreadPool **: 764**] ERROR com.hdx.routing.saf.SafUtils  - ** SAF THRESHOLD EXCEEDED ** currently SAF count is: 100 for Node : BJH/BJC/225/302/4.0 and route info: 

When we hit on this we need to search backwards over one min looking for the same ThreadPool ID in the error above it's : 764.

[9/26/17 13:45:18:675 EDT] 000215d9 SystemOut     O 4580329994 [SIBJMSRAThreadPool **: 764**] WARN com.hdx.routing.delivery.DeliveryEventHandlerSafV1  - **SAF** Failed sending to node 840153625 at TCPfalsefalse**64.46.236.20****10202**03ACK with RLogPK 

For this result we need to pull out the IP / Port and generate an alert. I have not extracted any fields yet.

We are still very new to Splunk. Thanks in advance for the help.

Carl

Tags (2)
0 Karma
1 Solution

DalJeanis
SplunkTrust
SplunkTrust

In splunk, it's almost always "wrong" to think in terms of taking one value and doing another search on that value.

Sometimes, when you have extracted fields, you can pull the values of the field in a subsearch and return it into a main search, but that way you are always going to the well at least twice for the water.

This way, you only have to go once, grab all the records you need, then copy what you need from the exceeded record to thelast record before it.

(index=foo "Threshold Exceeded" "225")
| eval TX = if(match(_raw,"THRESHOLD EXCEEDED") AND match(_raw,"225"),1,0)
| rex "SIBJMSRAThreadPool something something:\s+(?<ThreadPool>\d+)"  
| rex "something something (?<MyIP>\d+\.\d+\.\d+\.\d+) something (?<MyPort>\d+)something"
| fields TX ThreadPool MyIP MyPort

| rename COMMENT as "Get rid of everything that is not one or the other"
| where (TX=1 AND isnotnull(ThreadPool)) OR (isnotnull(ThreadPool) AND isnotnull(MyIP) AND isnotnull(MyPort))

| rename COMMENT as "Remember that these record were in reverse time order"
| rename COMMENT as "Copy threshold time back one record, keep only if it is within a minute"
| streamstats current=f last(eval(case(TX=1,_time))) as lastTX window=1 by ThreadPool
| where (TX=0) AND (_time>= lastTX - 60)   

| rename COMMENT as "Any record remaining is the last IP/Portrecord before a Threashold Exceeded record"

View solution in original post

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

In splunk, it's almost always "wrong" to think in terms of taking one value and doing another search on that value.

Sometimes, when you have extracted fields, you can pull the values of the field in a subsearch and return it into a main search, but that way you are always going to the well at least twice for the water.

This way, you only have to go once, grab all the records you need, then copy what you need from the exceeded record to thelast record before it.

(index=foo "Threshold Exceeded" "225")
| eval TX = if(match(_raw,"THRESHOLD EXCEEDED") AND match(_raw,"225"),1,0)
| rex "SIBJMSRAThreadPool something something:\s+(?<ThreadPool>\d+)"  
| rex "something something (?<MyIP>\d+\.\d+\.\d+\.\d+) something (?<MyPort>\d+)something"
| fields TX ThreadPool MyIP MyPort

| rename COMMENT as "Get rid of everything that is not one or the other"
| where (TX=1 AND isnotnull(ThreadPool)) OR (isnotnull(ThreadPool) AND isnotnull(MyIP) AND isnotnull(MyPort))

| rename COMMENT as "Remember that these record were in reverse time order"
| rename COMMENT as "Copy threshold time back one record, keep only if it is within a minute"
| streamstats current=f last(eval(case(TX=1,_time))) as lastTX window=1 by ThreadPool
| where (TX=0) AND (_time>= lastTX - 60)   

| rename COMMENT as "Any record remaining is the last IP/Portrecord before a Threashold Exceeded record"
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...