How do I use eval with a multivalued field in a transaction?

[ahmed]

Hello,

I am working with historical log data from a train system and I have two types of log files:

• log1: each row is an event that was logged every time a train arrived at a station. 
• log2: each row is an event that was logged every time a train station sign displayed a message. The messages predicted how many minutes it will take for the next train to arrive.

There are around 50 log2 events that correlate with each log1 event. I was able to group together all the log2 events with their corresponding log1 event into transactions. Here is the search I used to do this: 

sourcetype="log1" OR sourcetype="log2" 
| transaction serial platform maxspan=30m

This returns transactions which contain around 50 log2 events and 1 log1 event. How do I create a calculated field for each log2 event that makes up this transaction? The eval expression for the calculated field includes data from the log1 event in the transaction.

Here is how I tried to do this:

sourcetype="log1" OR sourcetype="log2"  
| transaction serial platform maxspan=30m 
| eval prediction_deviation = (arrival_date_time - (sign_date_time + next_min * 60))

"arrival_date_time" is a field from log1.

"sign_date_time" and "next_min" are fields from log2.

"prediction_deviation" is the calculated field which I am trying to add as a new column to all of the events from log2. 

When I run this command, only five values for "prediction_deviation" are calculated. I found out that this field is only being calculated for the transactions which only have one log2 event. These situations are outliers and there is no field being calculated for the rest of the transactions.  The eval command is only working when there is only one value for "sign_date_time" and "next_min". However, in most of the transactions there are about 50 values for these fields (one value for each log2 event in the transaction). 

How do I calculate the "prediction_deviation" for all of the log2 events in a transaction? The calculation of this field requires the "arrival_date_time" field for which there is only one value in each transaction.

Thank you for your help.

[ITWhisperer · Accepted solution]

Workout the predicted arrival time from the log2 entries before the transaction command, then you can use mvmap on the predicted arrival multivalue field, or mvexpand on the multivalue field to separate into different events.

[ahmed]

Thank you very much for your answer. 

The solution is to create a predicted_arrival calculated field for the log2 events and then use the mvexpand command to separate the multivalued field into separate events which you can use eval on. 

[to4kawa]

transaction command makes multi-value fields, so eval can't run appropriately.

Without a log, it's hard to say the rest.