All Apps and Add-ons

Lookup Editor: Won't save edits after update to Splunk (Cloud) 8.0.

wryanthomas
Contributor

My users are reporting their lookup edits aren't being saved since upgrade to Splunk Cloud 8.0 a few weeks ago. The error is "You do not have permission to perform this operation" when they do have access. (They can edit the lookups via inputlookup / outputlookup.)

I saw wiki troubleshooting doc:

   https://lukemurphey.net/projects/splunk-lookup-editor/wiki/Troubleshooting

When I execute the recommended query:

index=_internal (sourcetype="lookup_editor_controller" OR sourcetype=lookup_editor_rest_handler OR sourcetype=lookup_backups_rest_handler)

I see no ERROR events at all -- just INFO events. And I don't see any events (at all) since 6/8 -- which was three days after the upgrade to 8.x. 

Does that mean the REST function isn't working as of 6/8?

Labels (1)
Tags (1)
0 Karma
1 Solution

wryanthomas
Contributor

Thank you very much, Luke, for replying.

We *just* figured it out. (Splunk Support figured it out -- thanks to your documentation.)

In our Splunk 7.x (Splunk Cloud) instance (greenfield, 2019), we had removed upload_lookup_files capability from our "analyst" role (sort of in between user and power user). In Splunk 7.x -- our "analyst" role members were still able to use Lookup Editor. But when we upgraded to Splunk 8.x (late May), it stopped working for all users (non-admins) and we had no idea why. Since the version we were using in 7.x wasn't 8.x compatible, we had Splunk Support update it (which took weeks) to 3.4.1. But the update didn't solve the problem. Only then (Monday of this week) did real troubleshooting begin.

We added that capability to the analyst role this afternoon, and boom -- immediately, we're good again.

View solution in original post

0 Karma

pandacai
Engager

I ran into exactly the same problem after upgrading to Enterprise 8.1 this weekend -> and i see adding upload_lookup_files alone is not resolving my problem:( maybe there are other permissions needed?

0 Karma

wryanthomas
Contributor

@pandacai I'm eager to hear if/how you get this resolved. Our managed instance (Splunk Cloud) is getting upgraded to 8.1 on Dec 13. (Crossing fingers.)

0 Karma

wryanthomas
Contributor

The error users are getting is "You do not have permission to perform this operation".

But they do have access ... their workaround has been to use inputlookup and outputlookup to edit their lookups.  (I.e., access isn't the real problem.)

Here's a line from the users' HAR:

"content": {
"size": 85,
"mimeType": "text/plain",
"compression": 0,
"text": "{\"success\": false, \"message\": \"You do not have permission to perform this operation\"}"
},

0 Karma

wryanthomas
Contributor

Another tidbit from HAR that seems it might be relevant. http status response of 403. (I redacted cookie hashed values.)

"response": {
"status": 403,
"statusText": "Forbidden",
"httpVersion": "HTTP/1.1",
"headers": [
{
"name": "Cache-Control",
"value": "no-store, no-cache, must-revalidate, max-age=0"
},
{
"name": "Date",
"value": "Tue, 23 Jun 2020 14:20:59 GMT"
},
{
"name": "Expires",
"value": "Thu, 26 Oct 1978 00:00:00 GMT"
},
{
"name": "Server",
"value": "nginx"
},
{
"name": "Set-Cookie",
"value": "splunkd_8443=[...redacted...]; Path=/; Secure; HttpOnly; Max-Age=3600; Expires=Tue, 23 Jun 2020 15:20:59 GMT"
},
{
"name": "Set-Cookie",
"value": "splunkweb_csrf_token_8443=[...redacted...]; Path=/; Secure; Max-Age=157680000; Expires=Sun, 22 Jun 2025 14:20:59 GMT"
},
{
"name": "Strict-Transport-Security",
"value": "max-age=31536000; includeSubDomains"
},
{
"name": "Vary",
"value": "*"
},
{
"name": "X-Frame-Options",
"value": "SAMEORIGIN"
},
{
"name": "Content-Length",
"value": "85"
},
{
"name": "Connection",
"value": "keep-alive"
}
],

0 Karma

LukeMurphey
Champion

Thanks for the detailed analysis.

The backend has not changed in a while (7 months). So I tend to think the problem is on the frontend. Though I cannot explain why a 403 would happen.

Could you see if clearing cache would help? See https://community.splunk.com/t5/Archive/How-do-I-clear-the-cache-to-see-the-changes-after-updating-a...

I'm thinking that perhaps something on the front-end code is being cached and somehow makes the REST call is not getting created correctly. I did do a semi-dramatic replacement of a major front-end library in version 3.4.0 (UI doesn't look much different but the code has changed substantially). My best theory is either:

1. There is a bug related to the new library that the testing missed

2. The old version is being cached

0 Karma

wryanthomas
Contributor

Thank you very much, Luke, for replying.

We *just* figured it out. (Splunk Support figured it out -- thanks to your documentation.)

In our Splunk 7.x (Splunk Cloud) instance (greenfield, 2019), we had removed upload_lookup_files capability from our "analyst" role (sort of in between user and power user). In Splunk 7.x -- our "analyst" role members were still able to use Lookup Editor. But when we upgraded to Splunk 8.x (late May), it stopped working for all users (non-admins) and we had no idea why. Since the version we were using in 7.x wasn't 8.x compatible, we had Splunk Support update it (which took weeks) to 3.4.1. But the update didn't solve the problem. Only then (Monday of this week) did real troubleshooting begin.

We added that capability to the analyst role this afternoon, and boom -- immediately, we're good again.

0 Karma

LukeMurphey
Champion

Awesome, thanks for the feedback. Let me look into updating the error messages; I should make this clearer.

0 Karma

wryanthomas
Contributor

Although our end-user experience issue is resolved, I'm still curious why we're not seeing logs  from _internal in the sourcetypes you specify (lookup*) in documentation.  Any ideas on that one?  Even though Lookup Editor is working, it is still true that we see no events with those sourcetypes since 6/8/2020.

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...