Security

Search backwards matching on value in current search result

johnca00
New Member

Hello -

I have a logging event like this one. We are searching on "Threshold Exceeded" AND "225"

9/26/17 13:45:18:690 EDT] 000215d9 SystemOut     O 4580330012 [SIBJMSRAThreadPool **: 764**] ERROR com.hdx.routing.saf.SafUtils  - ** SAF THRESHOLD EXCEEDED ** currently SAF count is: 100 for Node : BJH/BJC/225/302/4.0 and route info: 

When we hit on this we need to search backwards over one min looking for the same ThreadPool ID in the error above it's : 764.

[9/26/17 13:45:18:675 EDT] 000215d9 SystemOut     O 4580329994 [SIBJMSRAThreadPool **: 764**] WARN com.hdx.routing.delivery.DeliveryEventHandlerSafV1  - **SAF** Failed sending to node 840153625 at TCPfalsefalse**64.46.236.20****10202**03ACK with RLogPK 

For this result we need to pull out the IP / Port and generate an alert. I have not extracted any fields yet.

We are still very new to Splunk. Thanks in advance for the help.

Carl

Tags (2)
0 Karma
1 Solution

DalJeanis
SplunkTrust
SplunkTrust

In splunk, it's almost always "wrong" to think in terms of taking one value and doing another search on that value.

Sometimes, when you have extracted fields, you can pull the values of the field in a subsearch and return it into a main search, but that way you are always going to the well at least twice for the water.

This way, you only have to go once, grab all the records you need, then copy what you need from the exceeded record to thelast record before it.

(index=foo "Threshold Exceeded" "225")
| eval TX = if(match(_raw,"THRESHOLD EXCEEDED") AND match(_raw,"225"),1,0)
| rex "SIBJMSRAThreadPool something something:\s+(?<ThreadPool>\d+)"  
| rex "something something (?<MyIP>\d+\.\d+\.\d+\.\d+) something (?<MyPort>\d+)something"
| fields TX ThreadPool MyIP MyPort

| rename COMMENT as "Get rid of everything that is not one or the other"
| where (TX=1 AND isnotnull(ThreadPool)) OR (isnotnull(ThreadPool) AND isnotnull(MyIP) AND isnotnull(MyPort))

| rename COMMENT as "Remember that these record were in reverse time order"
| rename COMMENT as "Copy threshold time back one record, keep only if it is within a minute"
| streamstats current=f last(eval(case(TX=1,_time))) as lastTX window=1 by ThreadPool
| where (TX=0) AND (_time>= lastTX - 60)   

| rename COMMENT as "Any record remaining is the last IP/Portrecord before a Threashold Exceeded record"

View solution in original post

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

In splunk, it's almost always "wrong" to think in terms of taking one value and doing another search on that value.

Sometimes, when you have extracted fields, you can pull the values of the field in a subsearch and return it into a main search, but that way you are always going to the well at least twice for the water.

This way, you only have to go once, grab all the records you need, then copy what you need from the exceeded record to thelast record before it.

(index=foo "Threshold Exceeded" "225")
| eval TX = if(match(_raw,"THRESHOLD EXCEEDED") AND match(_raw,"225"),1,0)
| rex "SIBJMSRAThreadPool something something:\s+(?<ThreadPool>\d+)"  
| rex "something something (?<MyIP>\d+\.\d+\.\d+\.\d+) something (?<MyPort>\d+)something"
| fields TX ThreadPool MyIP MyPort

| rename COMMENT as "Get rid of everything that is not one or the other"
| where (TX=1 AND isnotnull(ThreadPool)) OR (isnotnull(ThreadPool) AND isnotnull(MyIP) AND isnotnull(MyPort))

| rename COMMENT as "Remember that these record were in reverse time order"
| rename COMMENT as "Copy threshold time back one record, keep only if it is within a minute"
| streamstats current=f last(eval(case(TX=1,_time))) as lastTX window=1 by ThreadPool
| where (TX=0) AND (_time>= lastTX - 60)   

| rename COMMENT as "Any record remaining is the last IP/Portrecord before a Threashold Exceeded record"
0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...