Adding a field via lookup at index time

[oleg106]

Hello,

I've been trying to figure out the most efficient way to do this and a bit unclear on ingest-time vs automatic lookups or another way of this this.  This is a simple and probably a common use case:

Events are coming in with event_id code which is not friendly user.  I want to do a lookup at index time against the event_id code (integer) and add a field called event_id_desc with what that code resolves to in a lookup (e.g. event_id: 5, event_id_desc: user login).  What is the most efficient way of doing this?  There are 1500 static codes in the csv.

Thanks!  

[richgalloway · Accepted solution]

Why do the lookup at index time?  That will add to your storage costs and can slow down ingestion.

Have you considered an automatic lookup?  It will map event_id to event_id_desc at search time without the user/dashboard taking any explicit action.

[oleg106]

This is a relatively low velocity search, so I am not concerned about licensing here.  Would like to bake this into the log for better ES compatibility.  Would auto lookup be an issue in a distributed environment with ES?

[richgalloway]

ES does automatic lookups already so I would expect no issues.