Is the automatic lookup table used by the indexer?

[rxdeleon]

When an automatic lookup table is defined, is that used by the indexer to add the new fields or is it the search head that does that?

[skoelpin · Accepted solution]

It's the search head. The data is already indexed on the indexer so it would be a search time function

[rxdeleon]

Thanks, skoelpin, for the quick reply. If that's the case, does that mean that the raw data found by the indexer would all be shipped to the search head? And that's where the lookup table would be applied?

[skoelpin]

Partially correct. The SH will search the data on the indexer, the indexer will not ship its data to the SH

Data lives on the indexer and when a scheduled/ad-hoc search kicks off on the SH, the SH will search the data on the indexers and the automatic lookup logic will be applied at search time. A good way to think about this is, say you create an automatic lookup and want to change it after a day. You can easily change it because it's done on the fly at search time without baking any rules onto the indexers

[skoelpin]

@rxdeleon please accept the answer if I answered your question

[rxdeleon]

@skoelpin, I understand that the automatic lookup logic will be applied at search time. But which component does that? Is it the search head or the indexer? If it's the search head, then that means the search results, no matter how big, would be sent to the search head where the automatic lookup logic would be applied.

I would wish that it's the indexer that does it so that extracted fields could be used to filter out irrelevant events to minimize data being sent back to the search head (for performance reasons).

[skoelpin]

It's the search head. Lookups have always been a bottleneck which is why I always tell customers that you should use a stats before the lookup.

For index time lookups, you should check out Cribl. It integrates directly with Splunk! I had a long conversation with their CEO @clintsharp at CONF and was pretty impressed with the features it has.

https://blog.cribl.io/2018/09/17/enriching-data-in-motion-with-ingest-time-lookups/

[rxdeleon]

Thanks for the Cribl info, @skoelpin. I'll check it out.