Archive

Mapping data from small sourcetype to another larger sourcetype

New Member

Hello! I'm having this issue of merging data from one sourcetype to another larger sourcetype. Example:

index=ecs_internal (sourcetype=ecs:encode parcel_id=* earliest=-30d@d latest=@d) OR (sourcetype=ecs:input barcode_id=* earliest=-60d@d latest=@d)
| eval parcel_id=if(isnotnull(parcel_id), parcel_id, barcode_id)
| stats latest(*) as *, sum(eval(if(sourcetype=ecs:encode, 1, 0))) as valid by parcel_id
| where valid >= 1

Aim: Display all events in ecs:encode, then lookup latest related information from ecs:input. Basically, only 30% of all events in ecs:input would have data relevant to ecs:encode, thus my search is extremely slow... Join command would not work as I'm having millions of events in both sourcetypes.

Many thanks in advance!

0 Karma

Communicator

Isn't the phrase:

| where valid >=1

effectively saying don't show any stats that aren't in sourcetype=ecs:input?

0 Karma

New Member

Hi memarshall63, yes it works, but problem is the search takes too much time (currently it's taking me more than 1 hour!). Query above is just a simplified version, I have to do a lot of evals before the stats.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!