topic Why can I not expand lookup field due to a reference cycle in the lookup configuration? in Splunk Cloud Platform

Why can I not expand lookup field due to a reference cycle in the lookup configuration?

derekho55 — Fri, 18 Mar 2022 04:54:10 GMT

on Splunk Cloud (8.1.2101.1) I'm encountering a warning message in my search results - trying to figure out why this is popping up. Anybody have any idea what this message means and how to resolve it?

Re: Cannot expand lookup field due to a reference cycle in the lookup configuration.

threepointonefo — Thu, 18 Mar 2021 18:50:53 GMT

Not an answer, but I am seeing the same problem with the 'user' field in wineventlog data. But if I change my search to use the Logon_Account field instead then I get the same results but without the warning. (Though Logon_Account is only present because I'm just looking at records where EventCode=4776.)

There does not seem to be any problem the results, it is only a warning and not an error. But I would like to know for sure what is causing it.

I should add, we are using Splunk Cloud 8.1.2101.2

Re: Cannot expand lookup field due to a reference cycle in the lookup configuration.

jmyers — Wed, 17 Mar 2021 18:44:17 GMT

Same issue here as well, which didn't occur until 8.1.2101.2. I haven't been able to find anything to indicate what may have caused the change in the release notes. So far it's only happened on one field, I'll see if I can find any other fields it's happening with and if there's a pattern there.

Edit: check out your automatic field lookups. I had one that was self-referential for some reason, and after fixing that I am no longer seeing the error.

Re: Cannot expand lookup field due to a reference cycle in the lookup configuration.

jmyers — Wed, 17 Mar 2021 18:46:22 GMT

Do you have any automatic lookups involving the "action" field?

I had one for the field I was having an issue with, and for some reason the lookup was self-referential. Once I fixed it (in my case, copied it in case deleting it caused problems, then deleted it), I no longer had the error.

Re: Cannot expand lookup field due to a reference cycle in the lookup configuration.

threepointonefo — Thu, 18 Mar 2021 18:48:12 GMT

@jmyers

I think you're right about the automatic lookups. I have this in the search log

03-18-2021 16:35:08.065 INFO SearchEvaluatorBasedExpander [20232 searchOrchestrator] - Performing lookup expansions
03-18-2021 16:35:08.065 WARN AutoLookupDriver [20232 searchOrchestrator] - Detected a cycle: fieldname=UserId, visitedFields=UserId,user
03-18-2021 16:35:08.065 WARN AutoLookupDriver [20232 searchOrchestrator] - sid:1616085307.16530 Cannot expand lookup field 'user' due to a reference cycle in the lookup configuration. Rewrite the lookup configuration to remove the reference cycle.

But I've yet to find the lookup that is causing the problem.

Re: Cannot expand lookup field due to a reference cycle in the lookup configuration.

rayl — Tue, 13 Apr 2021 17:50:53 GMT

Right, this message was recently upgraded from DEBUG to WARN in order to surface the reference cycle issue that can slow down the search performance.

We are looking into whether we can make the message more actionable, but the other WARN message in search.log might be able to help locate the offending lookup(s). In this case, Splunk Software saw `user` field in the SPL, and figured that it could be mapped from `UserId` field through an automatic lookup. And `UserId` field itself could be mapped from `UserId` field itself, forming a reference cycle. So, most likely, there is a reference cycle like: UserId OUTPUT UserId, or, UserId_1 AS UserId OUTPUT UserId_2 AS UserId. Removing that cycle (which is not necessary) should get rid of the warning.

Re: Cannot expand lookup field due to a reference cycle in the lookup configuration.

threepointonefo — Wed, 14 Apr 2021 13:23:07 GMT

@rayl

Thanks for the background information. It helps to understand what's going on.
I no longer get the warning with the 'user' field in wineventlog data (I don't know why that problem has disappeared) but I do still get it on the 'UserId' field in office365.

With office365 data I find that with the time picker set to 30 days searches on the 'UserId' field take a very long time and scan millions of events whereas searches on the 'user' field take only a second or two and scan just hundreds of events. The two field names reference the same data so it makes a good comparison. However with all the various lookups created by the add-ons it is not easy to find what is causing the problem. I really need to know the names of the lookups involved. Is there any way of getting that information?

Thanks.

Re: Cannot expand lookup field due to a reference cycle in the lookup configuration.

bwheel — Thu, 27 May 2021 17:43:34 GMT

I found this issue myself, and some of the automatic lookups had things like:

windows_action_lookup Type OUTPUTNEW action, action AS status

Resaving them then resolved them back to
windows_action_lookup Type OUTPUTNEW action AS status

Which has then resolved the issue.
It appears that the Splunk_TA_Windows had these lookup issues, seems to work correctly as most entries had action fields in them anyway.

Re: Cannot expand lookup field due to a reference cycle in the lookup configuration.

heikothiel — Thu, 17 Jun 2021 07:35:54 GMT

This worked for me - thanks.

Re: Cannot expand lookup field due to a reference cycle in the lookup configuration.

johnhuang — Tue, 29 Jun 2021 20:27:36 GMT

Had this issue with "user", it was due to the Salesforce App and Addon. You can update it to the latest version which fixes it.

The easiest way to find out the cause:

1. Run search that has the issue.

2. View Job -> Inspect Job

3. Click the link "search.log" in the Inspect Job windows.

3. Search the text for "cycle".

Re: Cannot expand lookup field due to a reference cycle in the lookup configuration.

threepointonefo — Wed, 30 Jun 2021 11:37:22 GMT

@johnhuang

Thanks for your message. That explains why the problem with the 'user' field went away as I had updated the Salesforce App some time after finding the problem.
I still have the problem with 'UserId' in our Office 365 index though I can work around that by using the 'user' field which has the same data.
I too have been looking in search.log but I am unable to tell which lookup is causing the problem. I get "Reading schema for lookup table" for about every lookup we have and that is followed by:

06-30-2021 10:58:54.127 INFO SearchEvaluatorBasedExpander [14716 searchOrchestrator] - Performing lookup expansions
06-30-2021 10:58:54.128 WARN AutoLookupDriver [14716 searchOrchestrator] - Detected a cycle: fieldname=UserId, visitedFields=UserId
06-30-2021 10:58:54.128 WARN AutoLookupDriver [14716 searchOrchestrator] - sid:1625050733.210543 Cannot expand lookup field 'UserId' due to a reference cycle in the lookup configuration. Rewrite the lookup configuration to remove the reference cycle.

Is there a way to tell which lookup is causing the problem?

@rayl says above that Splunk are looking into whether they can make the message more actionable but at present I can't see a way to identify the problem lookup.

Re: Cannot expand lookup field due to a reference cycle in the lookup configuration.

radam2000 — Wed, 30 Jun 2021 14:20:47 GMT

I have the same issue with a query

Cannot expand lookup field 'action' due to a reference cycle in the lookup configuration. Rewrite the lookup configuration to remove the reference cycle.

Can I have more information on what a reference cycle actually means and how to find it??? I think the terminology of this error message could be tweaked to provide more information other than "reference cycle"

I am not querying windowseventlogs but actually querying firepower logs

thanks in advance?

Re: Cannot expand lookup field due to a reference cycle in the lookup configuration.

rayl — Wed, 30 Jun 2021 16:15:17 GMT

The enhancement has been released in 8.2.2105, where a message in search.log would tell which exact lookups caused the reference cycle. Have your stack upgraded to that version if it hasn't been done.

Re: Cannot expand lookup field due to a reference cycle in the lookup configuration.

Yemi_Splunk — Fri, 16 Jul 2021 19:57:15 GMT

Starting from 8.2.2, you can click on the "job", then "inspect job", then click on "search log" Do "Ctrl" Find; and type in "reference cycle", copy the highlighted texts into a notepad, and you can see the problematic lookup.

You also do not need to re-save this lookup (It is the Automatic Lookup in most cases). The problem is because the Lookup input field and Lookup output field has blank fields. Delete these blanks (Click on the Automatic Lookup name, and then delete the blank field line under lookup input fields and lookup output field) and click save.

It should be resolved.

Re: Cannot expand lookup field due to a reference cycle in the lookup configuration.

threepointonefo — Tue, 27 Jul 2021 13:40:37 GMT

@Yemi_SplunkThanks for your post on this. We are on Version: 8.2.2104.1

I've had a look at the logs again but I can't enough information to identify the lookup that is causing the problem.

The search that gives this error is searching for a specific UserId in Office 365 data. When I run the search I get the warning:

warn : Cannot expand lookup field 'UserId' due to a reference cycle in the lookup configuration. Rewrite the lookup configuration to remove the reference cycle.

And in search.log I get this:

07-27-2021 11:34:41.648 INFO  SearchEvaluatorBasedExpander [13577 searchOrchestrator] -  Performing lookup expansions
07-27-2021 11:34:41.648 WARN  AutoLookupDriver [13577 searchOrchestrator] - Detected a cycle: fieldname=UserId, visitedFields=UserId
07-27-2021 11:34:41.648 WARN  AutoLookupDriver [13577 searchOrchestrator] - sid:fe0545bbe5a575d7_tmp Cannot expand lookup field 'UserId' due to a reference cycle in the lookup configuration. Rewrite the lookup configuration to remove the reference cycle.
07-27-2021 11:34:41.648 INFO  SearchEvaluatorBasedExpander [13577 searchOrchestrator] -  Lookup expansion took 0 ms

I can't see the name of the problem lookup. Am I missing something obvious or doing something wrong?

@rayl I didn't see the post about 8.2.2105 until after I posted the above.

I eagerly await the update to our Splunk Cloud instance!

Re: Cannot expand lookup field due to a reference cycle in the lookup configuration.

radam2000 — Tue, 03 Aug 2021 20:20:35 GMT

Thanks rayl

I am on splunk cloud version 8.2.2104.1 and am scheduled for upgrade on August 11th ... will have to wait for the upgrade to find it...

Rich

Re: Cannot expand lookup field due to a reference cycle in the lookup configuration.

sean_wong — Thu, 12 Aug 2021 02:13:11 GMT

I have the same problem on my Splunk search head version 8.2.1 and it was resolved after the completion of below steps.

Error Messages: -

"Cannot expand lookup field 'action' due to a reference cycle in the lookup configuration. Check search .log for details and update the lookup configuration to remove the reference cycle."

1. Log onto search head by putty, run below command.

$SPLUNK_HOME/splunk btool props list --debug | grep -E 'LOOKUP.*\saction\s.*\saction\s'

Result:-

/opt/splunk/etc/apps/Splunk_TA_cisco-asa/default/props.conf LOOKUP-cisco_asa_action_lookup_1 = cisco_asa_action_lookup vendor_action as action OUTPUT action, action AS Cisco_ASA_action

2. Log onto to search head on web browser

3. Go to `Settings` - `Lookups` - `Automatic lookups`.
4. Look for a lookup named as `LOOKUP-cisco_asa_action_lookup_1`.

p.s. The captured screen is showing the last result (problem was solved). I have not regenerate the problem, please treat it as an example.

5. Click on 'cisco:asa : LOOKUP-cisco_asa_action_lookup_1'.
6. Upon opening the lookup in lookup editor, click `Save` without touching any field.
7. Repeat the steps`1.` to `4.` if there is another lookup has same issue.
8. After the above, run the command again and the new lookups should look like as followings.

/opt/splunk/etc/apps/Splunk_TA_cisco-asa/local/props.conf LOOKUP-cisco_asa_action_lookup_1 = cisco_asa_action_lookup vendor_action AS action OUTPUT action AS Cisco_ASA_action

9. Restart the search head once and see whether the problem is solved.

Sean Wong

Re: Cannot expand lookup field due to a reference cycle in the lookup configuration.

Saurabh_Goyal — Thu, 30 Sep 2021 02:15:17 GMT

Kindly review lookup - sfdc:loginhistory : LOOKUP-SFDC-USER_NAME

I tried saving the lookup, but it dint help.

Re: Cannot expand lookup field due to a reference cycle in the lookup configuration.

duartet — Wed, 13 Oct 2021 13:42:13 GMT

Worked for me!

The issue is indeed with the add-on Splunk_TA_cisco_asa on default/props.conf.

This is what is should look like, but it's fixed by adding to local/props.conf:

LOOKUP-cisco_asa_action_lookup_1 = cisco_asa_action_lookup vendor_action AS action OUTPUT action AS Cisco_ASA_action

Re: Cannot expand lookup field due to a reference cycle in the lookup configuration.

walsborn — Wed, 03 Nov 2021 17:39:08 GMT

Well Done Sean, resolved my issue as well.