Splunk Search

Mapping data from small sourcetype to another larger sourcetype

New Member

Hello! I'm having this issue of merging data from one sourcetype to another larger sourcetype. Example:

index=ecs_internal (sourcetype=ecs:encode parcel_id=* earliest=-30d@d latest=@d) OR (sourcetype=ecs:input barcode_id=* earliest=-60d@d latest=@d)
| eval parcel_id=if(isnotnull(parcel_id), parcel_id, barcode_id)
| stats latest(*) as *, sum(eval(if(sourcetype=ecs:encode, 1, 0))) as valid by parcel_id
| where valid >= 1

Aim: Display all events in ecs:encode, then lookup latest related information from ecs:input. Basically, only 30% of all events in ecs:input would have data relevant to ecs:encode, thus my search is extremely slow... Join command would not work as I'm having millions of events in both sourcetypes.

Many thanks in advance!

0 Karma


Isn't the phrase:

| where valid >=1

effectively saying don't show any stats that aren't in sourcetype=ecs:input?

0 Karma

New Member

Hi memarshall63, yes it works, but problem is the search takes too much time (currently it's taking me more than 1 hour!). Query above is just a simplified version, I have to do a lot of evals before the stats.

0 Karma
Get Updates on the Splunk Community!

Accelerate Service Onboarding, Decomposition, Troubleshooting - and more with ITSI’s ...

Accelerate Service Onboarding, Decomposition, Troubleshooting - and more! Faster Time to ValueManaging and ...

New Release | Splunk Enterprise 9.3

Admins and Analyst can benefit from:  Seamlessly route data to your local file system to save on storage ...

2024 Splunk Career Impact Survey | Earn a $20 gift card for participating!

Hear ye, hear ye! The time has come again for Splunk's annual Career Impact Survey!  We need your help by ...