Splunk Search

Mapping data from small sourcetype to another larger sourcetype

New Member

Hello! I'm having this issue of merging data from one sourcetype to another larger sourcetype. Example:

index=ecs_internal (sourcetype=ecs:encode parcel_id=* earliest=-30d@d latest=@d) OR (sourcetype=ecs:input barcode_id=* earliest=-60d@d latest=@d)
| eval parcel_id=if(isnotnull(parcel_id), parcel_id, barcode_id)
| stats latest(*) as *, sum(eval(if(sourcetype=ecs:encode, 1, 0))) as valid by parcel_id
| where valid >= 1

Aim: Display all events in ecs:encode, then lookup latest related information from ecs:input. Basically, only 30% of all events in ecs:input would have data relevant to ecs:encode, thus my search is extremely slow... Join command would not work as I'm having millions of events in both sourcetypes.

Many thanks in advance!

0 Karma


Isn't the phrase:

| where valid >=1

effectively saying don't show any stats that aren't in sourcetype=ecs:input?

0 Karma

New Member

Hi memarshall63, yes it works, but problem is the search takes too much time (currently it's taking me more than 1 hour!). Query above is just a simplified version, I have to do a lot of evals before the stats.

0 Karma
Get Updates on the Splunk Community!

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

Observability Newsletter Highlights | March 2023

 March 2023 | Check out the latest and greatestSplunk APM's New Tag Filter ExperienceSplunk APM has updated ...

Security Newsletter Updates | March 2023

 March 2023 | Check out the latest and greatestUnify Your Security Operations with Splunk Mission Control The ...