Splunk Search

To what degree does chaining (or nesting) automatic lookups actually work?

SplunkTrust
SplunkTrust

Example:
Say I have two lookups A and B. Let's say they're both file-based lookups (even though I don't think it actually matters)
A takes an input field called "first" and outputs an output field called "second"
B takes an input field called "second" and outputs an output field called "third"

Let's say they are both "automatic" courtesy of this in props.conf:

LOOKUP-aaa = A first OUTPUT second
LOOKUP-bbb = B second OUTPUT third

We've also made an example with no inline renaming or anything - the external field names happen to be identical to the column names in the lookup. It's a little artificial but hey it's an example.

More background

A) The Splunk docs claim that this isn't supported. ( https://docs.splunk.com/Documentation/Splunk/8.0.2/Knowledge/Makeyourlookupautomatic See the statement "Splunk software does not support nested automatic lookups")
However
2) They clearly do work at least a little, because apps use them and there are answers posts telling you how to do it.
( https://answers.splunk.com/answers/94609/automatic-lookup-on-a-field-that-is-automatically-looked-up...
https://answers.splunk.com/answers/209148/can-you-perform-an-automatic-lookup-based-on-the-o.html
)

Question 1 -- how much of this actually works?
Question 2 -- what changes have there been to that answer over time, if any?

1 Solution

SplunkTrust
SplunkTrust

Other answers are great. And I'll incorporate any comments, suggestions on this answer into this one as necessary.

However some research indicates that as long as:
1) you have named your LOOKUP-* key names such that when those key names are sorted alphabetically, they end up in the order they need to be for the output fields to flow into the input fields further down as needed.

AND then
2) EITHER
a) at least one output field from each level of automatic lookups involved, is named explicitly in your search language
or
b) You are in "verbose" mode.

Then it will work great!

Basically the lookups seem to get evaluated sequentially such that you can chain them together and the output fields will happily flow down and get used as the input fields for lower levels.

This seems to be true in standalone and in dist-search and when true when the lookups run on the SH and true when the lookups are out at the indexers.

HOWEVER - its the "required_fields" side of things that isn't implemented -- ie the part of Splunk's search language parser that automatically tries to optimize which field-extractions and lookup-definitions get run, does not do anything clever to "unwind" this chain. So you have to make sure that piece is done, either by explicitly mentioning the fields (forcing them all to get added to the field extractions that are happening), or by doing "verbose" mode which just tells splunk to run all the field extractions there are.

I've tested every major release back to 6.0.X and it works and the behavior seems identical. For what it's worth I'm pretty sure the same behavior goes back to 5.0 if not all the way back to 4.X

One Note - implicitly here I mean "within one single app". Although you could technically I think chain a lookup in app B to depend on the output field from another lookup in App A, that's just not a great idea as far as long term reliability and reasonable maintenance/supportability.

View solution in original post

SplunkTrust
SplunkTrust

Other answers are great. And I'll incorporate any comments, suggestions on this answer into this one as necessary.

However some research indicates that as long as:
1) you have named your LOOKUP-* key names such that when those key names are sorted alphabetically, they end up in the order they need to be for the output fields to flow into the input fields further down as needed.

AND then
2) EITHER
a) at least one output field from each level of automatic lookups involved, is named explicitly in your search language
or
b) You are in "verbose" mode.

Then it will work great!

Basically the lookups seem to get evaluated sequentially such that you can chain them together and the output fields will happily flow down and get used as the input fields for lower levels.

This seems to be true in standalone and in dist-search and when true when the lookups run on the SH and true when the lookups are out at the indexers.

HOWEVER - its the "required_fields" side of things that isn't implemented -- ie the part of Splunk's search language parser that automatically tries to optimize which field-extractions and lookup-definitions get run, does not do anything clever to "unwind" this chain. So you have to make sure that piece is done, either by explicitly mentioning the fields (forcing them all to get added to the field extractions that are happening), or by doing "verbose" mode which just tells splunk to run all the field extractions there are.

I've tested every major release back to 6.0.X and it works and the behavior seems identical. For what it's worth I'm pretty sure the same behavior goes back to 5.0 if not all the way back to 4.X

One Note - implicitly here I mean "within one single app". Although you could technically I think chain a lookup in app B to depend on the output field from another lookup in App A, that's just not a great idea as far as long term reliability and reasonable maintenance/supportability.

View solution in original post