Splunk Search

Transactions within transactions

tyronetv
Communicator

I have a set of two logs that share a common field (RID). One log contains the "user" actions while the other log contains the java, odbc, etc., actions (ibatis log).

What I need to do, is figure out if there is a delay between when a user makes a request and the actual connection to the database.

2013-08-19 06:40:32,904 DEBUG [andler-RMICallHandler-192] RID=1376919632894-63948 c.r.t.r.o.service.OnlineDecisioning - <DAO CALL START> {USER ID} null {DAO PROCEDURE} OnlineDecisioning.getTransactionDetails {INPUT PARAMS} {transactionid=2404611298, callinguserid=-5710326788253269740}

2013-08-19 06:40:32,908 DEBUG [andler-RMICallHandler-192] RID=1376919632894-63948 java.sql.Connection - {conn-473946} Connection

... some other actions .....

2013-08-19 06:40:33,332 DEBUG [andler-RMICallHandler-192] RID=1376919632894-63948 c.r.t.r.o.service.OnlineDecisioning - <DAO CALL START> {USER ID} null {DAO PROCEDURE} OnlineDecisioning.lockTransaction {INPUT PARAMS} {transactionid=2404611298, latestchangeid=449327573, callinguserid=-5710326788253269740}

2013-08-19 06:40:33,333 DEBUG [andler-RMICallHandler-192] RID=1376919632894-63948 java.sql.Connection - {conn-473948} Connection

...some other actions....

2013-08-19 06:45:06,130 DEBUG [andler-RMICallHandler-192] RID=1376919632894-63948 c.r.t.r.o.service.OnlineDecisioning - <DAO CALL START> {USER ID} null {DAO PROCEDURE} OnlineDecisioning.getCouponFieldMappings {INPUT PARAMS} {documentTypeId=2642998, callinguserid=-5710326788253269740}

2013-08-19 06:45:06,130 DEBUG [andler-RMICallHandler-192] RID=1376919632894-63948 java.sql.Connection - {conn-486974} Connection

... and other various versions of this....

2013-08-19 06:40:32,904 DEBUG [andler-RMICallHandler-192] RID=1376919632894-63948 c.r.t.r.o.service.OnlineDecisioning - <DAO CALL START> {USER ID} null {DAO PROCEDURE} OnlineDecisioning.getTransactionDetails {INPUT PARAMS} {transactionid=2404611298, callinguserid=-5710326788253269740}

2013-08-19 06:40:32,908 DEBUG [andler-RMICallHandler-192] RID=1376919632894-63948 java.sql.Connection - {conn-473946} Connection

... some other actions .....

2013-08-19 06:40:33,332 DEBUG [andler-RMICallHandler-192] RID=1376919632894-63948 c.r.t.r.o.service.OnlineDecisioning - <DAO CALL START> {USER ID} null {DAO PROCEDURE} OnlineDecisioning.lockTransaction {INPUT PARAMS} {transactionid=2404611298, latestchangeid=449327573, callinguserid=-5710326788253269740}

2013-08-19 06:40:33,333 DEBUG [andler-RMICallHandler-192] RID=1376919632894-63948 java.sql.Connection - {conn-473948} Connection

...some other actions....

2013-08-19 06:45:06,130 DEBUG [andler-RMICallHandler-192] RID=1376919632894-63948 c.r.t.r.o.service.OnlineDecisioning -<DAO CALL START> {USER ID} null {DAO PROCEDURE} OnlineDecisioning.getCouponFieldMappings {INPUT PARAMS} {documentTypeId=2642998, callinguserid=-5710326788253269740}

2013-08-19 06:45:06,130 DEBUG [andler-RMICallHandler-192] RID=1376919632894-63948 java.sql.Connection - {conn-486974} Connection

... and other various versions of this....

As you can see, the connection is VERY fast. Unfortunately, sometimes, it is not. When it is NOT we have a problem that is about ready to increase exponentially and cause some pain for our customers and, well, me.

What I want to do is put in place a 'real time' monitor that says the time between the line with <DAO CALL START> in a sourcetype of t2_app and the java.sql.Connection in the sourcetype of ibatis cant exceed X seconds or milliseconds. (And, also to be able to graph normal response times.)

I can do a <DAO START> to a <DAO COMPLETE> but that is too much. Just need to know from start to DB connect.

Suggestions?

Thanks!

Tags (2)
0 Karma
1 Solution

tyronetv
Communicator

transaction RID startswith="DAO CALL START" endswith=" Connection"

Yeah... I should of figured that out sooner.

Thanks for the assist though.

View solution in original post

0 Karma

tyronetv
Communicator

transaction RID startswith="DAO CALL START" endswith=" Connection"

Yeah... I should of figured that out sooner.

Thanks for the assist though.

0 Karma

Ayn
Legend

Perhaps something like

... | transaction RID startswith="DAO CALL START" endswith="java.sql.Connection" | search duration>1

?

0 Karma

Ayn
Legend

The startswith argument means transaction will begin a new transaction whenever it sees that string. What are your results when you use this?

0 Karma

tyronetv
Communicator

Doesn't work. Been there done that. 😞

As you can see from my example, there are multiple connections to the DB under one RID.

I need each one encapsulated and that search didn't do it for me.

0 Karma

tyronetv
Communicator

AFAICT from the logs, and because I am doing a transaction based upon the RID, the connect is just the next sequential step after the DAO START.

0 Karma

Ayn
Legend

How do you relate the db connection to a particular DAO START message? Or is it always sequential so you can count on that the first connection following DAO START is the connection that's related to it?

0 Karma
Get Updates on the Splunk Community!

Continuing Innovation & New Integrations Unlock Full Stack Observability For Your ...

You’ve probably heard the latest about AppDynamics joining the Splunk Observability portfolio, deepening our ...

Monitoring Amazon Elastic Kubernetes Service (EKS)

As we’ve seen, integrating Kubernetes environments with Splunk Observability Cloud is a quick and easy way to ...

Cloud Platform & Enterprise: Classic Dashboard Export Feature Deprecation

As of Splunk Cloud Platform 9.3.2408 and Splunk Enterprise 9.4, classic dashboard export features are now ...