Security

Search backwards matching on value in current search result

johnca00
New Member

Hello -

I have a logging event like this one. We are searching on "Threshold Exceeded" AND "225"

9/26/17 13:45:18:690 EDT] 000215d9 SystemOut     O 4580330012 [SIBJMSRAThreadPool **: 764**] ERROR com.hdx.routing.saf.SafUtils  - ** SAF THRESHOLD EXCEEDED ** currently SAF count is: 100 for Node : BJH/BJC/225/302/4.0 and route info: 

When we hit on this we need to search backwards over one min looking for the same ThreadPool ID in the error above it's : 764.

[9/26/17 13:45:18:675 EDT] 000215d9 SystemOut     O 4580329994 [SIBJMSRAThreadPool **: 764**] WARN com.hdx.routing.delivery.DeliveryEventHandlerSafV1  - **SAF** Failed sending to node 840153625 at TCPfalsefalse**64.46.236.20****10202**03ACK with RLogPK 

For this result we need to pull out the IP / Port and generate an alert. I have not extracted any fields yet.

We are still very new to Splunk. Thanks in advance for the help.

Carl

Tags (2)
0 Karma
1 Solution

DalJeanis
SplunkTrust
SplunkTrust

In splunk, it's almost always "wrong" to think in terms of taking one value and doing another search on that value.

Sometimes, when you have extracted fields, you can pull the values of the field in a subsearch and return it into a main search, but that way you are always going to the well at least twice for the water.

This way, you only have to go once, grab all the records you need, then copy what you need from the exceeded record to thelast record before it.

(index=foo "Threshold Exceeded" "225")
| eval TX = if(match(_raw,"THRESHOLD EXCEEDED") AND match(_raw,"225"),1,0)
| rex "SIBJMSRAThreadPool something something:\s+(?<ThreadPool>\d+)"  
| rex "something something (?<MyIP>\d+\.\d+\.\d+\.\d+) something (?<MyPort>\d+)something"
| fields TX ThreadPool MyIP MyPort

| rename COMMENT as "Get rid of everything that is not one or the other"
| where (TX=1 AND isnotnull(ThreadPool)) OR (isnotnull(ThreadPool) AND isnotnull(MyIP) AND isnotnull(MyPort))

| rename COMMENT as "Remember that these record were in reverse time order"
| rename COMMENT as "Copy threshold time back one record, keep only if it is within a minute"
| streamstats current=f last(eval(case(TX=1,_time))) as lastTX window=1 by ThreadPool
| where (TX=0) AND (_time>= lastTX - 60)   

| rename COMMENT as "Any record remaining is the last IP/Portrecord before a Threashold Exceeded record"

View solution in original post

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

In splunk, it's almost always "wrong" to think in terms of taking one value and doing another search on that value.

Sometimes, when you have extracted fields, you can pull the values of the field in a subsearch and return it into a main search, but that way you are always going to the well at least twice for the water.

This way, you only have to go once, grab all the records you need, then copy what you need from the exceeded record to thelast record before it.

(index=foo "Threshold Exceeded" "225")
| eval TX = if(match(_raw,"THRESHOLD EXCEEDED") AND match(_raw,"225"),1,0)
| rex "SIBJMSRAThreadPool something something:\s+(?<ThreadPool>\d+)"  
| rex "something something (?<MyIP>\d+\.\d+\.\d+\.\d+) something (?<MyPort>\d+)something"
| fields TX ThreadPool MyIP MyPort

| rename COMMENT as "Get rid of everything that is not one or the other"
| where (TX=1 AND isnotnull(ThreadPool)) OR (isnotnull(ThreadPool) AND isnotnull(MyIP) AND isnotnull(MyPort))

| rename COMMENT as "Remember that these record were in reverse time order"
| rename COMMENT as "Copy threshold time back one record, keep only if it is within a minute"
| streamstats current=f last(eval(case(TX=1,_time))) as lastTX window=1 by ThreadPool
| where (TX=0) AND (_time>= lastTX - 60)   

| rename COMMENT as "Any record remaining is the last IP/Portrecord before a Threashold Exceeded record"
0 Karma
Get Updates on the Splunk Community!

Introducing Ingest Actions: Filter, Mask, Route, Repeat

WATCH NOW Ingest Actions (IA) is the best new way to easily filter, mask and route your data in Splunk® ...

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...