Playbook format block not using filtered data

[michael3]

I'm using the Cisco FireAMP app to return the trajectory of an endpoint, and the data includes a list of all running tasks/files.  For my test there are 500 items returned, with 9 marked as 'Malicious'.  I'm trying to filter for those and write the details to a note.  But the note always contains all 500 items, not just the 9.

My filter block (filter_2) is this:

if get_device_trajectory_2:action_result.data.*.events.*.file.disposition == Malicious

My format block (format_3) is this:

%%
File Name: {0}
- File Path: {1}
- Hash: {2}
- Category: {4}
- Parent: {3}
%%

where each of the variables refer to the filter block e.g.:

0: filtered-data:filter_2:condition_1:get_device_trajectory_2:action_result.data.*.events.*.file.file_name
1: filtered-data:filter_2:condition_1:get_device_trajectory_2:action_result.data.*.events.*.file.file_path
2: filtered-data:filter_2:condition_1:get_device_trajectory_2:action_result.data.*.events.*.file.identity.sha256
3: filtered-data:filter_2:condition_1:get_device_trajectory_2:action_result.data.*.events.*.file.parent.file_name
4: filtered-data:filter_2:condition_1:get_device_trajectory_2:action_result.data.*.events.*.detection

Finally, I use a Utility block to add the note.  The Utility block contents reference the format block:

format_3:formatted_data.*

The debugger shows this when running the filter block:

Mar 25, 13:52:54 : filter_2() called
Mar 25, 13:52:54 : phantom.condition(): called with 1 condition(s) '[['get_device_trajectory_2:action_result.data.*.events.*.file.disposition', '==', 'Malicious']]', operator : 'or', scope: 'new'
Mar 25, 13:52:54 : phantom.get_action_results() called for action name: get_device_trajectory_2 action run id: 0 app_run_id: 0
Mar 25, 13:52:54 : phantom.condition(): condition 1 to evaluate: LHS: get_device_trajectory_2:action_result.data.*.events.*.file.disposition OPERATOR: == RHS: Malicious
Mar 25, 13:52:54 : phantom.condition(): condition loop: condition 1, 'None' '==' 'Malicious' => result:False
Mar 25, 13:52:54 : phantom.condition(): condition loop: condition 1, 'None' '==' 'Malicious' => result:False
Mar 25, 13:52:54 : phantom.condition(): condition loop: condition 1, 'None' '==' 'Malicious' => result:False
Mar 25, 13:52:54 : phantom.condition(): condition loop: condition 1, 'None' '==' 'Malicious' => result:False
Mar 25, 13:52:54 : phantom.condition(): condition loop: condition 1, 'Unknown' '==' 'Malicious' => result:False
Mar 25, 13:52:54 : phantom.condition(): condition loop: condition 1, 'None' '==' 'Malicious' => result:False
Mar 25, 13:52:54 : phantom.condition(): condition loop: condition 1, 'Unknown' '==' 'Malicious' => result:False
Mar 25, 13:52:54 : phantom.condition(): condition loop: condition 1, 'Unknown' '==' 'Malicious' => result:False
Mar 25, 13:52:54 : phantom.condition(): condition loop: condition 1, 'Clean' '==' 'Malicious' => result:False
Mar 25, 13:52:55 : phantom.condition(): condition loop: condition 1, 'Unknown' '==' 'Malicious' => result:False
Mar 25, 13:52:55 : phantom.condition(): condition loop: condition 1, 'Malicious' '==' 'Malicious' => result:True
Mar 25, 13:52:55 : phantom.condition(): condition loop: condition 1, 'Malicious' '==' 'Malicious' => result:True
Mar 25, 13:52:55 : phantom.condition(): condition loop: condition 1, 'Malicious' '==' 'Malicious' => result:True
Mar 25, 13:52:55 : phantom.condition(): condition loop: condition 1, 'Malicious' '==' 'Malicious' => result:True
Mar 25, 13:52:55 : phantom.condition(): condition loop: condition 1, 'Malicious' '==' 'Malicious' => result:True
Mar 25, 13:52:55 : phantom.condition(): condition loop: condition 1, 'Malicious' '==' 'Malicious' => result:True
Mar 25, 13:52:55 : phantom.condition(): condition loop: condition 1, 'Malicious' '==' 'Malicious' => result:True
Mar 25, 13:52:55 : phantom.condition(): condition loop: condition 1, 'Malicious' '==' 'Malicious' => result:True
Mar 25, 13:52:55 : phantom.condition(): condition loop: condition 1, 'Malicious' '==' 'Malicious' => result:True
Mar 25, 13:52:55 : phantom.condition(): condition loop: condition 1, 'Unknown' '==' 'Malicious' => result:False
Mar 25, 13:52:55 : phantom.condition(): condition loop: condition 1, 'Unknown' '==' 'Malicious' => result:False
Mar 25, 13:52:55 : phantom.condition(): condition loop: condition 1, 'None' '==' 'Malicious' => result:False
Mar 25, 13:52:55 : phantom.condition(): condition loop: condition 1, 'Unknown' '==' 'Malicious' => result:False

so it looks like it's correctly identifying the malicious files.  The debugger shows this when running the format block:

Mar 25, 13:52:55 : format_3() called
Mar 25, 13:52:55 : phantom.collect2(): called for datapath['filtered-data:filter_2:condition_1:get_device_trajectory_2:action_result.data.*.events.*.file.file_name'], scope: new and filter_artifacts: []
Mar 25, 13:52:55 : phantom.get_run_data() called for key filtered-data:filter_2:condition_1
Mar 25, 13:52:55 : phantom.collect2(): Classified datapaths as [<DatapathClassification.NAMED_FILTERED_ACTION_RESULT: 9>]
Mar 25, 13:52:55 : phantom.collect2(): called for datapath['filtered-data:filter_2:condition_1:get_device_trajectory_2:action_result.data.*.events.*.file.file_path'], scope: new and filter_artifacts: []
Mar 25, 13:52:55 : phantom.get_run_data() called for key filtered-data:filter_2:condition_1
Mar 25, 13:52:55 : phantom.collect2(): Classified datapaths as [<DatapathClassification.NAMED_FILTERED_ACTION_RESULT: 9>]
Mar 25, 13:52:55 : phantom.collect2(): called for datapath['filtered-data:filter_2:condition_1:get_device_trajectory_2:action_result.data.*.events.*.file.identity.sha256'], scope: new and filter_artifacts: []
Mar 25, 13:52:55 : phantom.get_run_data() called for key filtered-data:filter_2:condition_1
Mar 25, 13:52:55 : phantom.collect2(): Classified datapaths as [<DatapathClassification.NAMED_FILTERED_ACTION_RESULT: 9>]
Mar 25, 13:52:55 : phantom.collect2(): called for datapath['filtered-data:filter_2:condition_1:get_device_trajectory_2:action_result.data.*.events.*.file.parent.file_name'], scope: new and filter_artifacts: []
Mar 25, 13:52:55 : phantom.get_run_data() called for key filtered-data:filter_2:condition_1
Mar 25, 13:52:55 : phantom.collect2(): Classified datapaths as [<DatapathClassification.NAMED_FILTERED_ACTION_RESULT: 9>]
Mar 25, 13:52:55 : phantom.collect2(): called for datapath['filtered-data:filter_2:condition_1:get_device_trajectory_2:action_result.data.*.events.*.detection'], scope: new and filter_artifacts: []
Mar 25, 13:52:55 : phantom.get_run_data() called for key filtered-data:filter_2:condition_1
Mar 25, 13:52:56 : phantom.collect2(): Classified datapaths as [<DatapathClassification.NAMED_FILTERED_ACTION_RESULT: 9>]
Mar 25, 13:52:56 : save_run_data() saving 136.29 KB with key format_3:formatted_data_
Mar 25, 13:52:56 : save_run_data() saving 140.23 KB with key format_3__as_list:formatted_data_

there are 9 malicious files and it looks like that's what it's saying in the debugger, so again it seems like it's using the filtered data correctly.  

But my note always has 500 items in it.  I'm not sure what I'm doing wrong.  Can anyone offer any help, because I'm stuck.  Thanks.

[SOARt_of_Lost · Accepted solution]

Working with just this example, the same applies across the board.

get_device_trajectory_2:action_result.data.*.events.*.file.parent.file_name

.data.*.events.*. is most likely your problem. Every time your filter block hits a true, you're telling your format block to pull in all of the file names in the event data from get_device_trajectory_2. You'll need to find a way to tell it to only pull in the information from the index of the item you care about. Something like 

get_device_trajectory_2:action_result.data.*.events.X.file.parent.file_name​

where X is the item in the list that evaluated true.

[michael3]

Yes, this was it.  The filter wasn't able to deal with the multiple levels in my data.

I ended up replacing the filter with a code block that ran the same conditional statement and saved the positive matches to a new list (or, in my case, 5 lists for the 5 fields I needed).  Then I fed those lists into the format block instead.

Thanks for the help!

[michael3]

Quick update.  I changed the format block to use format_3:formatted_data instead of formatted_data.*.  The note looks a lot nicer, but it's still 500 items.