Splunk Search

Tracking multiple transactions


What might a query look like if your data is structured like:

.....several events from one or more log files   ......
Jun 18, 11:36:10 "Start Button Pressed"
.....several events from one or more log files ......
Jun 18, 11:38:12 "Stop Button Pressed"
...more events.....
Jun 18, 11:46:10 "Start Button Pressed"
.....several events......
Jun 18, 11:48:12 "Stop Button Pressed"
.....several events......

| eval EventType=case(match(Info, "^Start Button Pressed""),"start" ,match(Info, "^Stop Button Pressed"),"end") 
| search EventType="start" OR  EventType="end"

The problem with the above 'search' is that it filters out the events that are not Start or Stop.
I need to be able to display all events that occur between Start and Stop.

And I need to be able to guarantee that my data is sequential

| sort 0 d TimeStamp 

Thanks for any suggestions!

0 Karma

Esteemed Legend

Never use transaction. Try this:

index=YouShouldAlwaysSpecifyAnIndex AND sourctype=AndSourcetypeToo
| streamstats count(eval(match(Info, "^Stop Button Pressed")))) AS sessionID BY host
| reverse
| stats list(_raw) AS events BY host sessionID
0 Karma


I not having much luck with splunk. However I have been able to extract fields from the various logs but but have had no luck of correlating any of the data and identifying transactions within my data.

Because there are several teams that own the various software modules that comprise our payment solution I can not easily ask them to add a correlation token all of the myriad methods calls.

Below is some structure of data:

terminal.log - start
Jun 24, 14:39:17.889035, ..... data fields..... "StartPaymentActivity ....... PaymentId(3145735)"

several other logs * that are capturing events that are happening in between
the start and end.

2019-06-24 14:39:17,940 ..... data fields..... Info field captured in splunk

terminal.log - end
Jun 24, 14:40:33.704066, ..... data fields..... Info field captured --> "Done with Merchant transaction no paymentId!!."

Also note there are more then one transaction in these log files.

Now I do not fully understand splunk transaction or stats commands

But I have been trying to use rex with a startswith or endswith is this possible?

But notice the last two posts one say's use transaction and the next one say avoid it and use stats

The final output I need should look like:

sourcetype event _time _ paymentId Info -- Start Transaction 1
sourcetype event _time _ paymentId Info -- data events from different logs
sourcetype event _time _ paymentId Info -- End Transaction 1

sourcetype event _time _ paymentId Info -- Start Transaction n
sourcetype event _time _ paymentId Info -- data events from different logs
sourcetype event _time _ paymentId Info -- End Transaction n

Any guidance is appreciated.
Thanks in advance for any and all help.

0 Karma


All Transactions are sequential .... if there were overlaps this would have not possibility of working at all.

0 Karma

Esteemed Legend

Did you even try my solution? You can worry about understanding it later.

0 Karma


I could get your query to produce any results - here is my starting point:

index=* OR index=_* sourcetype=Terminal.debug OR sourcetype="PaymentGateway*" 
| eval Action =case(match(Info, "^ApplyPayment\(\) - ManagerId.+PaymentId"),"StartTran" ,
    match(Info, "^Done with  Merchant lines"),"EndTran", match(Info, "^Exiting"), "Exiting", 
    match(Info, ""), "Info", 1=1, Action) 
| eval EventTime=strfTime(if(in(Action, "StartTran", "EndTran"), _time,""),"%Y-%m-%d %H:%M:%S.%3N") 
| reverse 
| search Action= "StartTran" OR Action= "EndTran"  

## this is wrong I need to see what is between these to times....
| sort 0 _time
| stats ????
| table _time Action Info

0 Karma


I think the suggestion that uses streamstats has the assumption that your starts and stops are single threaded. Would your raw events ever contain situations where the starts/stops can overlap?


_time_0: Start Button - Transaction 1
_time_1: Start Button - Transaction 2
_time_2: Stop Button - Transaction 1
_time_3: Stop Button - Transaction 2

If there's no overlap, then I can see the streamstats approach working. However, if there is overlap, the transactions command use of 'someuniquefield' is what helps you.

(Note: I did not test either of these suggestions to be sure.)

Good luck.

0 Karma

Splunk Employee
Splunk Employee

You can use the transaction command with startswith and endswith:

| transaction someuniqefield startswith="info=Start Button Pressed" endswith="info=Stop Button Pressed"

someuniquefield should be a field that can be used to identify the transaction. This will group together events that have the same value of that field from Start Button Pressed to Stop Button Pressed.

0 Karma