topic Re: Tracking multiple transactions in Splunk Search

Tracking multiple transactions

dowdag — Tue, 18 Jun 2019 23:07:53 GMT

What might a query look like if your data is structured like:

.....several events from one or more log files   ......
Jun 18, 11:36:10 "Start Button Pressed"
.....several events from one or more log files ......
Jun 18, 11:38:12 "Stop Button Pressed"
...more events.....
Jun 18, 11:46:10 "Start Button Pressed"
.....several events......
Jun 18, 11:48:12 "Stop Button Pressed"
.....several events......

| eval EventType=case(match(Info, "^Start Button Pressed""),"start" ,match(Info, "^Stop Button Pressed"),"end") 
| search EventType="start" OR  EventType="end"

The problem with the above 'search' is that it filters out the events that are not Start or Stop.
I need to be able to display all events that occur between Start and Stop.

And I need to be able to guarantee that my data is sequential

| sort 0 d TimeStamp

Thanks for any suggestions!

Re: Tracking multiple transactions

kmorris_splunk — Wed, 19 Jun 2019 00:16:18 GMT

You can use the transaction command with startswith and endswith:

[YOUR BASE SEARCH HERE]
| transaction someuniqefield startswith="info=Start Button Pressed" endswith="info=Stop Button Pressed"

someuniquefield should be a field that can be used to identify the transaction. This will group together events that have the same value of that field from Start Button Pressed to Stop Button Pressed.

Re: Tracking multiple transactions

woodcock — Mon, 24 Jun 2019 00:15:00 GMT

Never use transaction. Try this:

index=YouShouldAlwaysSpecifyAnIndex AND sourctype=AndSourcetypeToo
| streamstats count(eval(match(Info, "^Stop Button Pressed")))) AS sessionID BY host
| reverse
| stats list(_raw) AS events BY host sessionID

Re: Tracking multiple transactions

dowdag — Wed, 26 Jun 2019 23:18:07 GMT

I not having much luck with splunk. However I have been able to extract fields from the various logs but but have had no luck of correlating any of the data and identifying transactions within my data.

Because there are several teams that own the various software modules that comprise our payment solution I can not easily ask them to add a correlation token all of the myriad methods calls.

Below is some structure of data:

terminal.log - start
Jun 24, 14:39:17.889035, ..... data fields..... "StartPaymentActivity ....... PaymentId(3145735)"

several other logs * that are capturing events that are happening in between
the start and end.

2019-06-24 14:39:17,940 ..... data fields..... Info field captured in splunk

terminal.log - end
Jun 24, 14:40:33.704066, ..... data fields..... Info field captured --> "Done with Merchant transaction no paymentId!!."

Also note there are more then one transaction in these log files.

Now I do not fully understand splunk transaction or stats commands

But I have been trying to use rex with a startswith or endswith is this possible?

But notice the last two posts one say's use transaction and the next one say avoid it and use stats

The final output I need should look like:

sourcetype event _time _ paymentId Info -- Start Transaction 1
sourcetype event _time _ paymentId Info -- data events from different logs
sourcetype event _time _ paymentId Info -- End Transaction 1

sourcetype event _time _ paymentId Info -- Start Transaction n
sourcetype event _time _ paymentId Info -- data events from different logs
sourcetype event _time _ paymentId Info -- End Transaction n

Any guidance is appreciated.
Thanks in advance for any and all help.

Re: Tracking multiple transactions

memarshall63 — Wed, 30 Sep 2020 01:04:07 GMT

I think the suggestion that uses streamstats has the assumption that your starts and stops are single threaded. Would your raw events ever contain situations where the starts/stops can overlap?

Like:

_time_0: Start Button - Transaction 1
_time_1: Start Button - Transaction 2
_time_2: Stop Button - Transaction 1
_time_3: Stop Button - Transaction 2

If there's no overlap, then I can see the streamstats approach working. However, if there is overlap, the transactions command use of 'someuniquefield' is what helps you.

(Note: I did not test either of these suggestions to be sure.)

Good luck.

Re: Tracking multiple transactions

dowdag — Thu, 27 Jun 2019 15:20:52 GMT

All Transactions are sequential .... if there were overlaps this would have not possibility of working at all.

Re: Tracking multiple transactions

woodcock — Thu, 27 Jun 2019 16:02:38 GMT

Did you even try my solution? You can worry about understanding it later.

Re: Tracking multiple transactions

dowdag — Thu, 27 Jun 2019 18:53:14 GMT

I could get your query to produce any results - here is my starting point:

index=* OR index=_* sourcetype=Terminal.debug OR sourcetype="PaymentGateway*" 
| eval Action =case(match(Info, "^ApplyPayment\(\) - ManagerId.+PaymentId"),"StartTran" ,
    match(Info, "^Done with  Merchant lines"),"EndTran", match(Info, "^Exiting"), "Exiting", 
    match(Info, ""), "Info", 1=1, Action) 
| eval EventTime=strfTime(if(in(Action, "StartTran", "EndTran"), _time,""),"%Y-%m-%d %H:%M:%S.%3N") 
| reverse 
| search Action= "StartTran" OR Action= "EndTran"

## this is wrong I need to see what is between these to times.... | sort 0 _time| stats ????
| table _time Action Info