Mapping data from small sourcetype to another larger sourcetype

[tleduc]

Hello! I'm having this issue of merging data from one sourcetype to another larger sourcetype. Example:

index=ecs_internal (sourcetype=ecs:encode parcel_id=* earliest=-30d@d latest=@d) OR (sourcetype=ecs:input barcode_id=* earliest=-60d@d latest=@d)
| eval parcel_id=if(isnotnull(parcel_id), parcel_id, barcode_id)
| stats latest(*) as *, sum(eval(if(sourcetype=ecs:encode, 1, 0))) as valid by parcel_id
| where valid >= 1

Aim: Display all events in ecs:encode, then lookup latest related information from ecs:input. Basically, only 30% of all events in ecs:input would have data relevant to ecs:encode, thus my search is extremely slow... Join command would not work as I'm having millions of events in both sourcetypes.

Many thanks in advance!

[memarshall63]

Isn't the phrase:

| where valid >=1

effectively saying don't show any stats that aren't in sourcetype=ecs:input?

[tleduc]

Hi memarshall63, yes it works, but problem is the search takes too much time (currently it's taking me more than 1 hour!). Query above is just a simplified version, I have to do a lot of evals before the stats.