I have the same problem on my Splunk search head version 8.2.1 and it was resolved after the completion of below steps.
Error Messages: -
"Cannot expand lookup field 'action' due to a reference cycle in the lookup configuration. Check search .log for details and update the lookup configuration to remove the reference cycle."
1. Log onto search head by putty, run below command.
$SPLUNK_HOME/splunk btool props list --debug | grep -E 'LOOKUP.*\saction\s.*\saction\s'
/opt/splunk/etc/apps/Splunk_TA_cisco-asa/default/props.conf LOOKUP-cisco_asa_action_lookup_1 = cisco_asa_action_lookup vendor_action as action OUTPUT action, action AS Cisco_ASA_action
2. Log onto to search head on web browser
3. Go to `Settings` - `Lookups` - `Automatic lookups`.
4. Look for a lookup named as `LOOKUP-cisco_asa_action_lookup_1`.
p.s. The captured screen is showing the last result (problem was solved). I have not regenerate the problem, please treat it as an example.
5. Click on 'cisco:asa : LOOKUP-cisco_asa_action_lookup_1'.
6. Upon opening the lookup in lookup editor, click `Save` without touching any field.
7. Repeat the steps`1.` to `4.` if there is another lookup has same issue.
8. After the above, run the command again and the new lookups should look like as followings.
/opt/splunk/etc/apps/Splunk_TA_cisco-asa/local/props.conf LOOKUP-cisco_asa_action_lookup_1 = cisco_asa_action_lookup vendor_action AS action OUTPUT action AS Cisco_ASA_action
9. Restart the search head once and see whether the problem is solved.
Worked for me!
The issue is indeed with the add-on Splunk_TA_cisco_asa on default/props.conf.
This is what is should look like, but it's fixed by adding to local/props.conf:
LOOKUP-cisco_asa_action_lookup_1 = cisco_asa_action_lookup vendor_action AS action OUTPUT action AS Cisco_ASA_action
I found this issue myself, and some of the automatic lookups had things like:
windows_action_lookup Type OUTPUTNEW action, action AS status
Resaving them then resolved them back to
windows_action_lookup Type OUTPUTNEW action AS status
Which has then resolved the issue.
It appears that the Splunk_TA_Windows had these lookup issues, seems to work correctly as most entries had action fields in them anyway.
Do you have any automatic lookups involving the "action" field?
I had one for the field I was having an issue with, and for some reason the lookup was self-referential. Once I fixed it (in my case, copied it in case deleting it caused problems, then deleted it), I no longer had the error.
Not an answer, but I am seeing the same problem with the 'user' field in wineventlog data. But if I change my search to use the Logon_Account field instead then I get the same results but without the warning. (Though Logon_Account is only present because I'm just looking at records where EventCode=4776.)
There does not seem to be any problem the results, it is only a warning and not an error. But I would like to know for sure what is causing it.
I should add, we are using Splunk Cloud 8.1.2101.2
Same issue here as well, which didn't occur until 8.1.2101.2. I haven't been able to find anything to indicate what may have caused the change in the release notes. So far it's only happened on one field, I'll see if I can find any other fields it's happening with and if there's a pattern there.
Edit: check out your automatic field lookups. I had one that was self-referential for some reason, and after fixing that I am no longer seeing the error.
I think you're right about the automatic lookups. I have this in the search log
03-18-2021 16:35:08.065 INFO SearchEvaluatorBasedExpander [20232 searchOrchestrator] - Performing lookup expansions
03-18-2021 16:35:08.065 WARN AutoLookupDriver [20232 searchOrchestrator] - Detected a cycle: fieldname=UserId, visitedFields=UserId,user
03-18-2021 16:35:08.065 WARN AutoLookupDriver [20232 searchOrchestrator] - sid:1616085307.16530 Cannot expand lookup field 'user' due to a reference cycle in the lookup configuration. Rewrite the lookup configuration to remove the reference cycle.
But I've yet to find the lookup that is causing the problem.
Right, this message was recently upgraded from DEBUG to WARN in order to surface the reference cycle issue that can slow down the search performance.
We are looking into whether we can make the message more actionable, but the other WARN message in search.log might be able to help locate the offending lookup(s). In this case, Splunk Software saw `user` field in the SPL, and figured that it could be mapped from `UserId` field through an automatic lookup. And `UserId` field itself could be mapped from `UserId` field itself, forming a reference cycle. So, most likely, there is a reference cycle like: UserId OUTPUT UserId, or, UserId_1 AS UserId OUTPUT UserId_2 AS UserId. Removing that cycle (which is not necessary) should get rid of the warning.
Thanks for the background information. It helps to understand what's going on.
I no longer get the warning with the 'user' field in wineventlog data (I don't know why that problem has disappeared) but I do still get it on the 'UserId' field in office365.
With office365 data I find that with the time picker set to 30 days searches on the 'UserId' field take a very long time and scan millions of events whereas searches on the 'user' field take only a second or two and scan just hundreds of events. The two field names reference the same data so it makes a good comparison. However with all the various lookups created by the add-ons it is not easy to find what is causing the problem. I really need to know the names of the lookups involved. Is there any way of getting that information?
Had this issue with "user", it was due to the Salesforce App and Addon. You can update it to the latest version which fixes it.
The easiest way to find out the cause:
1. Run search that has the issue.
2. View Job -> Inspect Job
3. Click the link "search.log" in the Inspect Job windows.
3. Search the text for "cycle".
Thanks for your message. That explains why the problem with the 'user' field went away as I had updated the Salesforce App some time after finding the problem.
I still have the problem with 'UserId' in our Office 365 index though I can work around that by using the 'user' field which has the same data.
I too have been looking in search.log but I am unable to tell which lookup is causing the problem. I get "Reading schema for lookup table" for about every lookup we have and that is followed by:
06-30-2021 10:58:54.127 INFO SearchEvaluatorBasedExpander [14716 searchOrchestrator] - Performing lookup expansions
06-30-2021 10:58:54.128 WARN AutoLookupDriver [14716 searchOrchestrator] - Detected a cycle: fieldname=UserId, visitedFields=UserId
06-30-2021 10:58:54.128 WARN AutoLookupDriver [14716 searchOrchestrator] - sid:1625050733.210543 Cannot expand lookup field 'UserId' due to a reference cycle in the lookup configuration. Rewrite the lookup configuration to remove the reference cycle.
Is there a way to tell which lookup is causing the problem?
@rayl says above that Splunk are looking into whether they can make the message more actionable but at present I can't see a way to identify the problem lookup.
I have the same issue with a query
Starting from 8.2.2, you can click on the "job", then "inspect job", then click on "search log" Do "Ctrl" Find; and type in "reference cycle", copy the highlighted texts into a notepad, and you can see the problematic lookup.
You also do not need to re-save this lookup (It is the Automatic Lookup in most cases). The problem is because the Lookup input field and Lookup output field has blank fields. Delete these blanks (Click on the Automatic Lookup name, and then delete the blank field line under lookup input fields and lookup output field) and click save.
It should be resolved.
@Yemi_SplunkThanks for your post on this. We are on Version: 8.2.2104.1
I've had a look at the logs again but I can't enough information to identify the lookup that is causing the problem.
The search that gives this error is searching for a specific UserId in Office 365 data. When I run the search I get the warning:
warn : Cannot expand lookup field 'UserId' due to a reference cycle in the lookup configuration. Rewrite the lookup configuration to remove the reference cycle.
And in search.log I get this:
07-27-2021 11:34:41.648 INFO SearchEvaluatorBasedExpander [13577 searchOrchestrator] - Performing lookup expansions 07-27-2021 11:34:41.648 WARN AutoLookupDriver [13577 searchOrchestrator] - Detected a cycle: fieldname=UserId, visitedFields=UserId 07-27-2021 11:34:41.648 WARN AutoLookupDriver [13577 searchOrchestrator] - sid:fe0545bbe5a575d7_tmp Cannot expand lookup field 'UserId' due to a reference cycle in the lookup configuration. Rewrite the lookup configuration to remove the reference cycle. 07-27-2021 11:34:41.648 INFO SearchEvaluatorBasedExpander [13577 searchOrchestrator] - Lookup expansion took 0 ms
I can't see the name of the problem lookup. Am I missing something obvious or doing something wrong?
@rayl I didn't see the post about 8.2.2105 until after I posted the above.
I eagerly await the update to our Splunk Cloud instance!