Hello! I'm having this issue of merging data from one sourcetype to another larger sourcetype. Example:
index=ecsinternal (sourcetype=ecs:encode parcelid=* earliest=-30d@d latest=@d) OR (sourcetype=ecs:input barcodeid=* earliest=-60d@d latest=@d)
| eval parcelid=if(isnotnull(parcelid), parcelid, barcodeid)
| stats latest(*) as *, sum(eval(if(sourcetype=ecs:encode, 1, 0))) as valid by parcelid
| where valid >= 1
Aim: Display all events in ecs:encode, then lookup latest related information from ecs:input. Basically, only 30% of all events in ecs:input would have data relevant to ecs:encode, thus my search is extremely slow... Join command would not work as I'm having millions of events in both sourcetypes.
Hi memarshall63, yes it works, but problem is the search takes too much time (currently it's taking me more than 1 hour!). Query above is just a simplified version, I have to do a lot of evals before the stats.