How to efficiently match events to lookup with wildcards across multiple fields (prefer most specific match)?

[tomapatan]

I'm working with a CSV lookup  that contains multiple fields which may include wildcard (*) values.
The lookup is structured such that some rows are very specific and others are generic (e.g. *, *, *, HOST, *). I want to enrich events from my base search with the best matching Offset (name of the field) from the lookup.

Challenges:

• Using lookup definition with match_type=WILDCARD(...) only works well if there’s a unique match — but in my case, I need to evaluate multiple potential matches and choose the most specific one.

• Using | map works correctly, but it's too slow.

[livehybrid]

Hi @tomapatan 

If you structure your lookups so that the more generic match is lower down the lookup than your more specific match, and you have your "Max Matches" set to 1 then it should match the more specific value first, else match the more generic one if not found.

For example - this is my test lookup:

You can see the more specific values are at the top.

I have configured a lookup definition with WILDCARD matches and a max matches = 1

Then I run a search, if country/town isnt set I am setting to "Unknown" but it could be any value.

It maps to 999 because this is the generic value for host1 if town/country is not set.

If I now set the country=UK:

I get a more specific value returned because it matches country=UK town=*

If I do host=host999 it matches host* in the lookup and I get an interestingField value of GHI:

Remember that you have to pass all the fields you want to match on to the lookup command, and you should have the more generic matches lower down the lookup file.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing