Archive

Mapping data from small sourcetype to another larger sourcetype

New Member

Hello! I'm having this issue of merging data from one sourcetype to another larger sourcetype. Example:

index=ecsinternal (sourcetype=ecs:encode parcelid=* earliest=-30d@d latest=@d) OR (sourcetype=ecs:input barcodeid=* earliest=-60d@d latest=@d)
| eval parcel
id=if(isnotnull(parcelid), parcelid, barcodeid)
| stats latest(*) as *, sum(eval(if(sourcetype=ecs:encode, 1, 0))) as valid by parcel
id
| where valid >= 1

Aim: Display all events in ecs:encode, then lookup latest related information from ecs:input. Basically, only 30% of all events in ecs:input would have data relevant to ecs:encode, thus my search is extremely slow... Join command would not work as I'm having millions of events in both sourcetypes.

Many thanks in advance!

0 Karma

Communicator

Isn't the phrase:

| where valid >=1

effectively saying don't show any stats that aren't in sourcetype=ecs:input?

0 Karma

New Member

Hi memarshall63, yes it works, but problem is the search takes too much time (currently it's taking me more than 1 hour!). Query above is just a simplified version, I have to do a lot of evals before the stats.

0 Karma