Why can I not expand lookup field due to a reference cycle in the lookup configuration?

[derekho55]

on Splunk Cloud (8.1.2101.1) I'm encountering a warning message in my search results - trying to figure out why this is popping up. Anybody have any idea what this message means and how to resolve it? 

[brenner]

In case anyone runs into the exact problem I had

error:

Detected a reference cycle in lookups: fields = sourcetype <-- sourcetype; lookups = [ cisco:prime : LOOKUP-prime_alarms_field_description

Turns out this was an issue with the mentioned automatic lookup.  If you go to settings, automatic lookups, and search for it you can pull it up.  In my case this had "sourcetype" listed as an input field and an output field so I removed it from the output field and now we can search any sourcetype again.

[sean_wong]

I have the same problem on my Splunk search head version 8.2.1 and it was resolved after the completion of below steps.

Error Messages: -

"Cannot expand lookup field 'action' due to a reference cycle in the lookup configuration. Check search .log for details and update the lookup configuration to remove the reference cycle."

1.  Log onto search head by putty, run below command.

$SPLUNK_HOME/splunk btool props list --debug | grep -E 'LOOKUP.*\saction\s.*\saction\s'

Result:- 

/opt/splunk/etc/apps/Splunk_TA_cisco-asa/default/props.conf LOOKUP-cisco_asa_action_lookup_1 = cisco_asa_action_lookup vendor_action as action OUTPUT action, action AS Cisco_ASA_action

2.  Log onto to search head on web browser

3.  Go to `Settings` - `Lookups` - `Automatic lookups`.
4.  Look for a lookup named as `LOOKUP-cisco_asa_action_lookup_1`.

p.s. The captured screen is showing the last result (problem was solved).  I have not regenerate the problem, please treat it as an example. 

5.  Click on 'cisco:asa : LOOKUP-cisco_asa_action_lookup_1'.
6.  Upon opening the lookup in lookup editor, click `Save` without touching any field.
7.  Repeat the steps`1.` to `4.` if there is another lookup has same issue.
8. After the above, run the command again and the new lookups should look like as followings.

/opt/splunk/etc/apps/Splunk_TA_cisco-asa/local/props.conf LOOKUP-cisco_asa_action_lookup_1 = cisco_asa_action_lookup vendor_action AS action OUTPUT action AS Cisco_ASA_action

9.  Restart the search head once and see whether the problem is solved.

Sean Wong

[anasfaruqui1]

works for me ....

clicked on all of the automatic lookup and save them one by one.

Thanks everyone!!!

[walsborn]

Well Done Sean, resolved my issue as well.

[duartet]

Worked for me!

The issue is indeed with the add-on Splunk_TA_cisco_asa on default/props.conf.

This is what is should look like, but it's fixed by adding to local/props.conf:

LOOKUP-cisco_asa_action_lookup_1 = cisco_asa_action_lookup vendor_action AS action OUTPUT action AS Cisco_ASA_action

[FrankVl]

But that fix actually removes some functionality, where the original "raw value" of the action field is matched against the vendor_action column in the lookup and then a normalized action value (from the action column) is returned, overwriting the action field. That normalization is critical for CIM datamodels and related searches/dashboards to work properly.

The real solution is that the TA should extract the raw action value into vendor_action and then use the lookup to translate it to action. Instead of the bizarre mess of evals and lookups with like 4 different action fields that is used currently.

[bwheel]

I found this issue myself, and some of the automatic lookups had things like:

windows_action_lookup Type OUTPUTNEW action, action AS status

Resaving them then resolved them back to 
windows_action_lookup Type OUTPUTNEW action AS status

Which has then resolved the issue.
It appears that the Splunk_TA_Windows had these lookup issues, seems to work correctly as most entries had action fields in them anyway.

[heikothiel]

This worked for me - thanks. 

[jmyers]

Do you have any automatic lookups involving the "action" field?

I had one for the field I was having an issue with, and for some reason the lookup was self-referential. Once I fixed it (in my case, copied it in case deleting it caused problems, then deleted it), I no longer had the error.

[threepointonefo]

Not an answer, but I am seeing the same problem with the 'user' field in wineventlog data. But if I change my search to use the Logon_Account field instead then I get the same results but without the warning. (Though Logon_Account is only present because I'm just looking at records where EventCode=4776.)

There does not seem to be any problem the results, it is only a warning and not an error. But I would like to know for sure what is causing it.

I should add, we are using Splunk Cloud 8.1.2101.2

[jmyers]

Same issue here as well, which didn't occur until 8.1.2101.2. I haven't been able to find anything to indicate what may have caused the change in the release notes. So far it's only happened on one field, I'll see if I can find any other fields it's happening with and if there's a pattern there. 

Edit: check out your automatic field lookups. I had one that was self-referential for some reason, and after fixing that I am no longer seeing the error.

[threepointonefo]

@jmyers

I think you're right about the automatic lookups. I have this in the search log

03-18-2021 16:35:08.065 INFO SearchEvaluatorBasedExpander [20232 searchOrchestrator] - Performing lookup expansions
03-18-2021 16:35:08.065 WARN AutoLookupDriver [20232 searchOrchestrator] - Detected a cycle: fieldname=UserId, visitedFields=UserId,user
03-18-2021 16:35:08.065 WARN AutoLookupDriver [20232 searchOrchestrator] - sid:1616085307.16530 Cannot expand lookup field 'user' due to a reference cycle in the lookup configuration. Rewrite the lookup configuration to remove the reference cycle.

But I've yet to find the lookup that is causing the problem.

[rayl]

Right, this message was recently upgraded from DEBUG to WARN in order to surface the reference cycle issue that can slow down the search performance.

We are looking into whether we can make the message more actionable, but the other WARN message in search.log might be able to help locate the offending lookup(s). In this case, Splunk Software saw `user` field in the SPL, and figured that it could be mapped from `UserId` field through an automatic lookup. And `UserId` field itself could be mapped from `UserId` field itself, forming a reference cycle. So, most likely, there is a reference cycle like: UserId OUTPUT UserId, or, UserId_1 AS UserId OUTPUT UserId_2 AS UserId. Removing that cycle (which is not necessary) should get rid of the warning. 

[threepointonefo]

@rayl 

Thanks for the background information. It helps to understand what's going on.
I no longer get the warning with the 'user' field in wineventlog data (I don't know why that problem has disappeared) but I do still get it on the 'UserId' field in office365.

With office365 data I find that with the time picker set to 30 days searches on the 'UserId' field take a very long time and scan millions of events whereas searches on the 'user' field take only a second or two and scan just hundreds of events. The two field names reference the same data so it makes a good comparison. However with all the various lookups created by the add-ons it is not easy to find what is causing the problem. I really need to know the names of the lookups involved. Is there any way of getting that information?

Thanks.

[johnhuang]

Had this issue with "user", it was due to the Salesforce App and Addon. You can update it to the latest version which fixes it.

The easiest way to find out the cause:

1. Run search that has the issue.

2. View Job -> Inspect Job

3. Click the link "search.log" in the Inspect Job windows.

3. Search the text for "cycle".

[threepointonefo]

@johnhuang 

Thanks for your message. That explains why the problem with the 'user' field went away as I had updated the Salesforce App some time after finding the problem.
I still have the problem with 'UserId' in our Office 365 index though I can work around that by using the 'user' field which has the same data.
I too have been looking in search.log but I am unable to tell which lookup is causing the problem. I get "Reading schema for lookup table" for about every lookup we have and that is followed by:

06-30-2021 10:58:54.127 INFO SearchEvaluatorBasedExpander [14716 searchOrchestrator] - Performing lookup expansions
06-30-2021 10:58:54.128 WARN AutoLookupDriver [14716 searchOrchestrator] - Detected a cycle: fieldname=UserId, visitedFields=UserId
06-30-2021 10:58:54.128 WARN AutoLookupDriver [14716 searchOrchestrator] - sid:1625050733.210543 Cannot expand lookup field 'UserId' due to a reference cycle in the lookup configuration. Rewrite the lookup configuration to remove the reference cycle.

Is there a way to tell which lookup is causing the problem?

@rayl  says above that Splunk are looking into whether they can make the message more actionable but at present I can't see a way to identify the problem lookup.

[radam2000]

I have the same issue with a query

Cannot expand lookup field 'action' due to a reference cycle in the lookup configuration. Rewrite the lookup configuration to remove the reference cycle.

 

Can I have more information on what a reference cycle actually means and how to find it??? I think the terminology of this error message could be tweaked to provide more information other than "reference cycle"

I am not querying windowseventlogs but actually querying firepower logs 

 

thanks in advance?

[Yemi_Splunk]

Starting from 8.2.2, you can click on the "job", then "inspect job", then click on "search log" Do "Ctrl" Find; and type in "reference cycle", copy the highlighted texts into a notepad, and you can see the problematic lookup.

You also do not need to re-save this lookup (It is the Automatic Lookup in most cases). The problem is because the Lookup input field and Lookup output field has blank fields. Delete these blanks (Click on the Automatic Lookup name, and then delete the blank field line under lookup input fields and lookup output field) and click save. 

It should be resolved.

[threepointonefo]

@Yemi_SplunkThanks for your post on this. We are on Version: 8.2.2104.1

I've had a look at the logs again but I can't enough information to identify the lookup that is causing the problem.

The search that gives this error is searching for a specific UserId in Office 365 data. When I run the search I get the warning:

warn : Cannot expand lookup field 'UserId' due to a reference cycle in the lookup configuration. Rewrite the lookup configuration to remove the reference cycle.

And in search.log I get this:

07-27-2021 11:34:41.648 INFO  SearchEvaluatorBasedExpander [13577 searchOrchestrator] -  Performing lookup expansions
07-27-2021 11:34:41.648 WARN  AutoLookupDriver [13577 searchOrchestrator] - Detected a cycle: fieldname=UserId, visitedFields=UserId
07-27-2021 11:34:41.648 WARN  AutoLookupDriver [13577 searchOrchestrator] - sid:fe0545bbe5a575d7_tmp Cannot expand lookup field 'UserId' due to a reference cycle in the lookup configuration. Rewrite the lookup configuration to remove the reference cycle.
07-27-2021 11:34:41.648 INFO  SearchEvaluatorBasedExpander [13577 searchOrchestrator] -  Lookup expansion took 0 ms

I can't see the name of the problem lookup. Am I missing something obvious or doing something wrong?

@rayl  I didn't see the post about 8.2.2105 until after I posted the above.

I eagerly await the update to our Splunk Cloud instance!

[rayl]

The enhancement has been released in 8.2.2105, where a message in search.log would tell which exact lookups caused the reference cycle. Have your stack upgraded to that version if it hasn't been done.