topic Re: Finding uncompleted transactions in Splunk Search

Finding uncompleted transactions

GratefulDude — Thu, 18 Feb 2010 05:04:50 GMT

I have application logs that will create a log when a user makes a request like:

2010-02-17 16:13:28.515 host1:1111:application DBG User made a requst[99999-1]: FOO (12345)

It then creates another log when the request is acknowledged like:

2010-02-17 16:13:29.118 host1:1111:application DBG reply for user 12345: request acknowledged

I am able to do a search and group both logs into pairs with transaction:

host="host1" source="C:\\logs\app*" ("DBG User made a request" OR "request acknowledged") | rex "DBG User made a requst: Foo \((?<ID>\d+)\) \[" | rex "DBG reply for user (?<ID>\d+): " | transaction ID maxspan=60s startswith="DBG User made a request" endswith="request acknowledged"

and I get a nice list of all the request/acknowledge pairs grouped together. What I need is to find (and alert) when I get a request, but not a matching acknowledge.

Any ideas?

Re: Finding uncompleted transactions

gkanapathy — Thu, 18 Feb 2010 09:27:00 GMT

You should just be able to add keepevicted=true to the transaction command options, then search on evicted=1:

... | transaction keepevicted=true ... | where evicted=1

http://docs.splunk.com/Documentation/Splunk/5.0/SearchReference/Transaction

Re: Finding uncompleted transactions

ziegfried — Fri, 19 Feb 2010 23:12:59 GMT

did this change for splunk 4.1? The evicted field doesn't seem to be part of the result. I had success with "… | transaction … | where closed_txn=0" though.

Re: Finding uncompleted transactions

gkanapathy — Sat, 20 Feb 2010 13:19:04 GMT

Hmm, looks like it changed at some point, you're right that in 4.0.9 the field is "closed_txn" and is the inverse of evicted, i.e., it's 1 for completed transactions.

Re: Finding uncompleted transactions

dianbo_1 — Tue, 25 May 2010 09:29:50 GMT

The startswith and endswith are "eventtype=A" and "eventtype=B" in my definition. But I just get those transactions just have end event (eventtype=B), and it can not display those just have start event(eventtype=A).

For example, if i do the search "eventtype=A | transaction router ip startswith="eventtype=A" endswith="eventtype=B" keepevicted=true", i should get many uncompleted transactions, but i get none.

Any ideas?

Thanks, Dianbo.

Re: Finding uncompleted transactions

Ledion_Bitincka — Sat, 12 Jun 2010 04:51:31 GMT

This is an outstanding issue (SPL-31786) scheduled to be fixed in out next maintenance release (4.1.4)

The following search might do what you want (if ID is a unique id at least within the 60 seconds that the transactions are supposed to last):

host="host1" source="C:\\logs\app*" ("DBG User made a request" OR "request acknowledged") | rex "DBG User made a requst: Foo \((?<ID>\d+)\) \[" | rex "DBG reply for user (?<ID>\d+): " | transaction ID maxspan=60s startswith="DBG User made a request" | search NOT "request acknowledged"

Re: Finding uncompleted transactions

GratefulDude — Tue, 15 Jun 2010 02:23:28 GMT

Thanks. What worked for me was doing a |search linecount<2 from my results. That matches all of my "requests" and "answers" up into transactions that should always be 2 lines.