I found this issue myself, and some of the automatic lookups had things like:
windows_action_lookup Type OUTPUTNEW action, action AS status
Resaving them then resolved them back to
windows_action_lookup Type OUTPUTNEW action AS status
Which has then resolved the issue.
It appears that the Splunk_TA_Windows had these lookup issues, seems to work correctly as most entries had action fields in them anyway.
Do you have any automatic lookups involving the "action" field?
I had one for the field I was having an issue with, and for some reason the lookup was self-referential. Once I fixed it (in my case, copied it in case deleting it caused problems, then deleted it), I no longer had the error.
Not an answer, but I am seeing the same problem with the 'user' field in wineventlog data. But if I change my search to use the Logon_Account field instead then I get the same results but without the warning. (Though Logon_Account is only present because I'm just looking at records where EventCode=4776.)
There does not seem to be any problem the results, it is only a warning and not an error. But I would like to know for sure what is causing it.
I should add, we are using Splunk Cloud 8.1.2101.2
Same issue here as well, which didn't occur until 8.1.2101.2. I haven't been able to find anything to indicate what may have caused the change in the release notes. So far it's only happened on one field, I'll see if I can find any other fields it's happening with and if there's a pattern there.
Edit: check out your automatic field lookups. I had one that was self-referential for some reason, and after fixing that I am no longer seeing the error.
I think you're right about the automatic lookups. I have this in the search log
03-18-2021 16:35:08.065 INFO SearchEvaluatorBasedExpander [20232 searchOrchestrator] - Performing lookup expansions
03-18-2021 16:35:08.065 WARN AutoLookupDriver [20232 searchOrchestrator] - Detected a cycle: fieldname=UserId, visitedFields=UserId,user
03-18-2021 16:35:08.065 WARN AutoLookupDriver [20232 searchOrchestrator] - sid:1616085307.16530 Cannot expand lookup field 'user' due to a reference cycle in the lookup configuration. Rewrite the lookup configuration to remove the reference cycle.
But I've yet to find the lookup that is causing the problem.
Right, this message was recently upgraded from DEBUG to WARN in order to surface the reference cycle issue that can slow down the search performance.
We are looking into whether we can make the message more actionable, but the other WARN message in search.log might be able to help locate the offending lookup(s). In this case, Splunk Software saw `user` field in the SPL, and figured that it could be mapped from `UserId` field through an automatic lookup. And `UserId` field itself could be mapped from `UserId` field itself, forming a reference cycle. So, most likely, there is a reference cycle like: UserId OUTPUT UserId, or, UserId_1 AS UserId OUTPUT UserId_2 AS UserId. Removing that cycle (which is not necessary) should get rid of the warning.
Thanks for the background information. It helps to understand what's going on.
I no longer get the warning with the 'user' field in wineventlog data (I don't know why that problem has disappeared) but I do still get it on the 'UserId' field in office365.
With office365 data I find that with the time picker set to 30 days searches on the 'UserId' field take a very long time and scan millions of events whereas searches on the 'user' field take only a second or two and scan just hundreds of events. The two field names reference the same data so it makes a good comparison. However with all the various lookups created by the add-ons it is not easy to find what is causing the problem. I really need to know the names of the lookups involved. Is there any way of getting that information?