Roles won't display in add/edit user/role page, "Failed to fetch data: Not Found"

[twinspop]

In the web_services.log file I see this error at the same time:

2013-11-14 17:46:44,232 ERROR   [528552d33521c6990] eai:164 - Failed to fetch dynamic element content from the server for splunkSource:/authentication/roles
[HTTP 404] https://127.0.0.1:8089/servicesNS/admin/launcher/authentication/roles?count=-1; [{'type': 'ERROR', 'code': None, 'text': 'Not Found'}]

The failure shows with red letters above the role list boxes: "Failed to fetch data: Not Found"

Any ideas where to start? I had not seen this problem before, but i hadn't added a role in some time. Upgraded from 5.0.2 -> 6.0 recently. Pooled search heads. 64-bit SUSE linux.

[lksridhar]

Hi Twinspop,

Can u please explain the above resolution steps?

[twinspop]

This situation really only applies if you are running a Search Head Pool. Which you shouldn't be doing. 🙂 The issue came about because SHP stores your Apps in a different location. So when I was upgrading splunk, it was updating the included apps in the default location, not the one actually used. Using tar to unpack the apps directory on top of the appropriate location should work: tar -xzf splunkforwarder-6.5.0-59c8927def0f-FreeBSD9-amd64.tgz --strip-components=2 -C /opt/splunkforwarder/etc splunkforwarder/etc/apps

[lksridhar]

Thanks Twinspop for your information, we are facing the issue in standalone search head , could provide the solution for this.

[twinspop · Accepted solution]

Bad upgrade procedure was the problem. With Seaarch Head Pooling, the default Splunk apps are stored in the shared storage area, not $splunk_home. I failed to realize this, which meant i never upgraded the default apps. Cleaned that up, and voila! Everything is working again.

[twinspop]

I had to merge the directories from the NEW tar file's splunk/etc/apps into my pooled servers' shared etc/apps directory. I ended up untarring into a tmp directory, cd'd into splunk/etc, created a new tar of apps, and finally untarred that new bundle in the shared dir's etc directory.

[ctapia]

Can you please explain how you have cleaned that up?