Splunk Search

Finding uncompleted transactions

GratefulDude
Explorer

I have application logs that will create a log when a user makes a request like:

2010-02-17 16:13:28.515 host1:1111:application DBG User made a requst[99999-1]: FOO (12345)

It then creates another log when the request is acknowledged like:

2010-02-17 16:13:29.118 host1:1111:application DBG reply for user 12345: request acknowledged

I am able to do a search and group both logs into pairs with transaction:

host="host1" source="C:\\logs\app*" ("DBG User made a request" OR "request acknowledged") | rex "DBG User made a requst: Foo \((?<ID>\d+)\) \[" | rex "DBG reply for user (?<ID>\d+): " | transaction ID maxspan=60s startswith="DBG User made a request" endswith="request acknowledged"

and I get a nice list of all the request/acknowledge pairs grouped together. What I need is to find (and alert) when I get a request, but not a matching acknowledge.

Any ideas?

Tags (1)

Ledion_Bitincka
Splunk Employee
Splunk Employee

This is an outstanding issue (SPL-31786) scheduled to be fixed in out next maintenance release (4.1.4)

The following search might do what you want (if ID is a unique id at least within the 60 seconds that the transactions are supposed to last):

host="host1" source="C:\\logs\app*" ("DBG User made a request" OR "request acknowledged") | rex "DBG User made a requst: Foo \((?<ID>\d+)\) \[" | rex "DBG reply for user (?<ID>\d+): " | transaction ID maxspan=60s startswith="DBG User made a request" | search NOT "request acknowledged"
0 Karma

GratefulDude
Explorer

Thanks. What worked for me was doing a |search linecount<2 from my results. That matches all of my "requests" and "answers" up into transactions that should always be 2 lines.

0 Karma

dianbo_1
Path Finder

The startswith and endswith are "eventtype=A" and "eventtype=B" in my definition. But I just get those transactions just have end event (eventtype=B), and it can not display those just have start event(eventtype=A).

For example, if i do the search "eventtype=A | transaction router ip startswith="eventtype=A" endswith="eventtype=B" keepevicted=true", i should get many uncompleted transactions, but i get none.

Any ideas?

Thanks, Dianbo.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

You should just be able to add keepevicted=true to the transaction command options, then search on evicted=1:

... | transaction keepevicted=true ... | where evicted=1

http://docs.splunk.com/Documentation/Splunk/5.0/SearchReference/Transaction

gkanapathy
Splunk Employee
Splunk Employee

Hmm, looks like it changed at some point, you're right that in 4.0.9 the field is "closed_txn" and is the inverse of evicted, i.e., it's 1 for completed transactions.

ziegfried
Influencer

did this change for splunk 4.1? The evicted field doesn't seem to be part of the result. I had success with "… | transaction … | where closed_txn=0" though.

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...