Transaction not grouping all events

[d1rtym0nk3y]

I'm working with data from various sip devices and trying to tie together various stages of a call going through our network into a transaction. each event has an incoming and outgoing call id (ic_call_id and og_call_id)

I've created an multi-value field call_id_combined from ic_call_id and og_call_id to be the basis of the transaction

... | transaction call_id_combined mvlist=t
| table _time ic_call_id og_call_id 

This gets most of the legs of the call, but skips legs where the og_call_id is a 'dead-end' This usually happens when the outbound connections fails or is refused

Heres an example from the above search

https:// dl.dropboxusercontent.com/u/9494424/splunk-ss-87678.png
(not enough karma to link or embed it..)

I've labelled them in the order they should appear, in the screenshot the timestamps are different, but this happens on events that all occur within the same second too.

If I anchor a transaction with startswith=(ic_ip_local=0) then the orphaned legs don't appear at all.

What can I do to pull these events together correctly?

Update

the full search is just prefixed with index=cdr as call_id_combined is setup as a search time extraction. but for the sake of this post, without using the extracted field it would look like this

index=cdr           
| eval call_id_combined=ic_call_id.",".og_call_id                    
| makemv call_id_combined delim=","                  
| transaction call_id_combined mvlist=t
| table _time, ic_call_id, og_call_id

the raw values for ic_call_id and og_call_id look like this

ic_call_id                          og_call_id
1359426042_75320729@<ip removed>    93704262-18259@<ip removed>     # inbound call
93704262-18259@<ip removed>         93704415-20024@<ip removed>     # route 1
93704415-20024@<ip removed>         93704429-18260@<ip removed>     # internal hop
93704262-18259@<ip removed>         93705988-20028@<ip removed>     # route 2 
93704262-18259@<ip removed>         93707998-20034@<ip removed>     # route 3

As you can see you can walk the legs of the calls from og_call_id to the next legs ic_call_id, but if that call leg fails you sometimes have to backtrack and then move on. Given the search above this call returns a transaction consisting only the first 3 events.

[wpreston · Accepted solution]

The only way I've found of doing this (at least it worked in a small sample I created) is to use transaction more than once. Try this and see if it works for your test cases:

index=cdr 
| transaction ic_call_id 
| eval new_call_id=mvappend(ic_call_id,og_call_id) 
| transaction new_call_id

Hopefully someone else will figure out a way to do this with only one transaction call.

[d1rtym0nk3y]

thanks, this does seem to work, although I can't get the events to appear in the correct order inside each transaction. A step in the right direction though

[d1rtym0nk3y]

i've updated the question, thanks for looking

[emechler_splunk]

Can you share the rest of the search, please? The relevant portions of the raw events would be helpful, too.