Splunk Search

Splunk Search & Reporting app returns "500 internal server error", only for one user.

LCM_BRogerson
Path Finder

I'm running Splunk Enterprise v 6.6.1 on Windows 2008 R2 (not by choice). Without making any configuration changes (that I'm aware of) one user has started receiving "500 internal server errors" when trying to access the Search & Reporting app. Other apps are not presenting this issue. All other users are fine. The errors are only present when UserA opens the Search & Reporting app.

The error message links to a search for index=_internal source=web_service.log requestid=[\xx]. When looking at the log file web_service.log in notepad++, there is no matching request id.
splunkd_acces.log is not showing any errors. All the entries for 127.0.0.1 with UserA have http status 200
There are entries in splunkd_ui_access.log and web_access.log with the HTTP 500 error and matching username and timestamp, but they useful for finding the problem. They only show the GET request, user-Agent, HTTP status, and request ID (web_acces) or session ID (splunkd_ui_access).
127.0.0.1 - [username] [date&time] "GET /en-US/app/search/search HTTP/1.1" 500 3037 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36" - [ID] 707ms

Restarting splunkd and the user's hosts have not had any impact. Threat/Socket limit is well above what we actually use, and if they were exhausted I would expect to see errors in splunkd.log and for all users to be seeing http 500 erros.

Has anyone else experienced an issue like this? Are there any log files other than those in [$SPLUNK_HOME$]\var\log\splunk that could help?

1 Solution

xavierashe
Contributor

The solution was moving the user's folder from [$SPLUNK_HOME$]/etc/user to a temp folder. They lost all of their objects, but the problem was resolved. If you want to try to save their objects, you could reassign them before clearing out their directory.

View solution in original post

0 Karma

xavierashe
Contributor

The solution was moving the user's folder from [$SPLUNK_HOME$]/etc/user to a temp folder. They lost all of their objects, but the problem was resolved. If you want to try to save their objects, you could reassign them before clearing out their directory.

0 Karma

LCM_BRogerson
Path Finder

To add to this.

One of the user's knowledge objects appears to be the cause of the errors. Reassigning them all may cause problems for the new account. Confirm there are no issues for the replacement account before removing the old account.

xavierashe
Contributor

Have you tried deleting the user's directory in [$SPLUNK_HOME$]/etc/users?

LCM_BRogerson
Path Finder

I've not tried that. Do you know how that would affect the knowledge object the user has created?

I assume they'll be orphaned and can then be reassigned to the new account?

0 Karma

xavierashe
Contributor

yes. Also, instead of deleting it, just move it to a temp folder.

0 Karma

LCM_BRogerson
Path Finder

Removed the user folder and created a new user with the same name.
User was able to log in and wasn't seeing the HTTP 500 errors. However their objects were not orphaned, just gone.
Manually restored the objects from their savedsearches.conf and local.meta files and then 500 errors returned, so I would assume there's something funky with one of their objects.

They only have a handfull of objects, the user can recreate them on their own. The SPL is all available in the .conf files.

Thank you for all your help!

xavierashe
Contributor

I'll put it in the answer below for other folks to find.

0 Karma

niketn
Legend

@LCM_BRogerson, what kind of role does the user have (admin/power or user)? Do other users also belong to the same role as this user?

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"

LCM_BRogerson
Path Finder

The user is part of the admin role. There are a few other users with the same role and none of them have issues.

0 Karma

xavierashe
Contributor

Since other admins seem to work fine, I doubt this is it, but figured I'd mention it just in case. I took these roles away from my users, and my admins started getting error 500s

rest_apps_view
rest_properties_get
rest_properties_set

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...