Why ES threat list lookup fails on pivot data, but works with direct search on indexed data?

[dragoslungu]

Is there anything different when running a lookup on data returned by a pivot compared to the same lookup running on data returned from a direct search on indexed data ?

The following lookup returns 1400 matches :

tag=ids tag=attack | lookup threatlist_lookup_by_str key AS dest
OUTPUTNEW name AS dest_threatlist_name| search dest_threatlist_name=*

If the same lookup runs agains a pivot, there are no matches:

| tstats `summariesonly` count from datamodel=Intrusion_Detection where
*  by _time,IDS_Attacks.src,IDS_Attacks.dest span=10m | `drop_dm_object_name("IDS_Attacks")` | lookup threatlist_lookup_by_str key AS dest OUTPUTNEW name AS dest_threatlist_name| search dest_threatlist_name=*

The root element definition of the Intrusion_Detection datamodel is the same as the direct search

tag=ids tag=attack

The purpose of this is to integrate custom threat lists in the default ES content like the "Threat List Activity" dashboard. Most of the ES default panels use datamodel / pivot searches which causes the lookups to fail.

Thanks.

[sjohnson_splunk]

Since your tstats is using summariesonly (which is true), it's possible that your data model accelerations are not keeping up and there are no results in the time window.