Splunk Search

Transaction not grouping subsequent filter strings

Naren26
Path Finder

I am trying to find the occurrence whenever the state changes due to the error. Below are my sample events:

2021/08/01 07:12:12.098 host=12345 In
2021/08/01 07:13:12.098 host=12345 In
2021/08/01 07:14:12.098 host=12345 Out
2021/08/01 07:15:12.098 host=12345 Out
2021/08/01 07:16:12.098 host=12345 In
2021/08/01 07:17:12.098 host=12345 In
2021/08/01 07:18:12.098 host=12345 Out
2021/08/01 07:18:35.098 host=12345 ERROR
2021/08/01 07:19:12.098 host=12345 In
2021/08/01 07:20:12.098 host=12345 Out

I need to group the events when the state (In/Out) changed due an ERROR event. For the above sample events, I should not get any result. Because, when the ERROR event happened, the host is already in "Out" stage. We need to monitor only when a "In" host changes to "Out" due to an ERROR.

I tried the below search

 

index=myindex ("Cut-In" OR "Cut-Out" OR "ERROR")
| rex "host=(?<host>\d+) (?<State>.*)"
| transaction host startswith="State=In" endswith="Out" maxspan=24h
| where searchmatch("ERROR")
| table _time host

 

But the above query returns a result by grouping the "In" state which logged at "07:16:12" as start of the transaction and "07:20:12" as end of the transaction. This is not a valid scenario.

Please help me in framing the logic.

Labels (2)
0 Karma

Naren26
Path Finder

This is not working too. I am still getting the event which should not be picked

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Given that the runanywhere example works, how does your real data differ from the example?

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Try something like this - note that 07:13 - 07:14 appears as a transaction and 07:12 - 07:15 is an outer transaction which may or may not be what you want, but given this is dummy data, this may just be a quirk of the example you made up.

| makeresults 
| eval _raw="2021/08/01 07:12:12.098 host=12345 In
2021/08/01 07:13:12.098 host=12345 In
2021/08/01 07:14:12.098 host=12345 Out
2021/08/01 07:15:12.098 host=12345 Out
2021/08/01 07:16:12.098 host=12345 In
2021/08/01 07:17:12.098 host=12345 In
2021/08/01 07:18:12.098 host=12345 Out
2021/08/01 07:18:35.098 host=12345 ERROR
2021/08/01 07:19:12.098 host=12345 In
2021/08/01 07:20:12.098 host=12345 Out"
| multikv noheader=t 
| table _raw



| rex "(?<datetime>.*)\shost=(?<host>\d+)\s(?<State>.*)"
| eval _time=strptime(datetime,"%Y/%m/%d %H:%M:%S.%Q")
| sort 0 - _time
| transaction host startswith="State=In" endswith="Out" maxspan=24h
0 Karma
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...