Lookup Editor: Won't save edits after update to Splunk (Cloud) 8.0.

[wryanthomas]

My users are reporting their lookup edits aren't being saved since upgrade to Splunk Cloud 8.0 a few weeks ago. The error is "You do not have permission to perform this operation" when they do have access. (They can edit the lookups via inputlookup / outputlookup.)

I saw wiki troubleshooting doc:

   https://lukemurphey.net/projects/splunk-lookup-editor/wiki/Troubleshooting

When I execute the recommended query:

index=_internal (sourcetype="lookup_editor_controller" OR sourcetype=lookup_editor_rest_handler OR sourcetype=lookup_backups_rest_handler)

I see no ERROR events at all -- just INFO events. And I don't see any events (at all) since 6/8 -- which was three days after the upgrade to 8.x. 

Does that mean the REST function isn't working as of 6/8?

[pandacai]

I ran into exactly the same problem after upgrading to Enterprise 8.1 this weekend -> and i see adding upload_lookup_files alone is not resolving my problem:( maybe there are other permissions needed?

[wryanthomas]

@pandacai I'm eager to hear if/how you get this resolved. Our managed instance (Splunk Cloud) is getting upgraded to 8.1 on Dec 13. (Crossing fingers.)

[wryanthomas]

The error users are getting is "You do not have permission to perform this operation".

But they do have access ... their workaround has been to use inputlookup and outputlookup to edit their lookups.  (I.e., access isn't the real problem.)

Here's a line from the users' HAR:

"content": {
"size": 85,
"mimeType": "text/plain",
"compression": 0,
"text": "{\"success\": false, \"message\": \"You do not have permission to perform this operation\"}"
},

[wryanthomas]

Another tidbit from HAR that seems it might be relevant. http status response of 403. (I redacted cookie hashed values.)

"response": {
"status": 403,
"statusText": "Forbidden",
"httpVersion": "HTTP/1.1",
"headers": [
{
"name": "Cache-Control",
"value": "no-store, no-cache, must-revalidate, max-age=0"
},
{
"name": "Date",
"value": "Tue, 23 Jun 2020 14:20:59 GMT"
},
{
"name": "Expires",
"value": "Thu, 26 Oct 1978 00:00:00 GMT"
},
{
"name": "Server",
"value": "nginx"
},
{
"name": "Set-Cookie",
"value": "splunkd_8443=[...redacted...]; Path=/; Secure; HttpOnly; Max-Age=3600; Expires=Tue, 23 Jun 2020 15:20:59 GMT"
},
{
"name": "Set-Cookie",
"value": "splunkweb_csrf_token_8443=[...redacted...]; Path=/; Secure; Max-Age=157680000; Expires=Sun, 22 Jun 2025 14:20:59 GMT"
},
{
"name": "Strict-Transport-Security",
"value": "max-age=31536000; includeSubDomains"
},
{
"name": "Vary",
"value": "*"
},
{
"name": "X-Frame-Options",
"value": "SAMEORIGIN"
},
{
"name": "Content-Length",
"value": "85"
},
{
"name": "Connection",
"value": "keep-alive"
}
],

[LukeMurphey]

Thanks for the detailed analysis.

The backend has not changed in a while (7 months). So I tend to think the problem is on the frontend. Though I cannot explain why a 403 would happen.

Could you see if clearing cache would help? See https://community.splunk.com/t5/Archive/How-do-I-clear-the-cache-to-see-the-changes-after-updating-a...

I'm thinking that perhaps something on the front-end code is being cached and somehow makes the REST call is not getting created correctly. I did do a semi-dramatic replacement of a major front-end library in version 3.4.0 (UI doesn't look much different but the code has changed substantially). My best theory is either:

1. There is a bug related to the new library that the testing missed

2. The old version is being cached

[wryanthomas · Accepted solution]

Thank you very much, Luke, for replying.

We *just* figured it out. (Splunk Support figured it out -- thanks to your documentation.)

In our Splunk 7.x (Splunk Cloud) instance (greenfield, 2019), we had removed upload_lookup_files capability from our "analyst" role (sort of in between user and power user). In Splunk 7.x -- our "analyst" role members were still able to use Lookup Editor. But when we upgraded to Splunk 8.x (late May), it stopped working for all users (non-admins) and we had no idea why. Since the version we were using in 7.x wasn't 8.x compatible, we had Splunk Support update it (which took weeks) to 3.4.1. But the update didn't solve the problem. Only then (Monday of this week) did real troubleshooting begin.

We added that capability to the analyst role this afternoon, and boom -- immediately, we're good again.

[LukeMurphey]

Awesome, thanks for the feedback. Let me look into updating the error messages; I should make this clearer.

[wryanthomas]

Although our end-user experience issue is resolved, I'm still curious why we're not seeing logs  from _internal in the sourcetypes you specify (lookup*) in documentation.  Any ideas on that one?  Even though Lookup Editor is working, it is still true that we see no events with those sourcetypes since 6/8/2020.