After upgrading my Splunk cluster to 6.3.1, I'm facing "500 Internal Server Error" all the time, after logging in to any of my Splunk instances (search head, deployment server ...)
My splunkd.log is full with lines like this:
WARN HttpListener - Can't handle request for /services/broker/connect/<peer id>, max thread limit for REST HTTP server is 2729, threads already in use is 2729
My web_service.log ends with things like this when the issue happens:
2015-11-30 08:21:49,790 DEBUG [565c071d9a4b27ac7a58] cplogging:55 - [30/Nov/2015:08:21:49] HTTP Traceback (most recent call last):
File "E:\Splunk\Python-2.7\Lib\site-packages\cherrypy_cprequest.py", line 606, in respond
cherrypy.response.body = self.handler()
File "E:\Splunk\Python-2.7\Lib\site-packages\cherrypy_cpdispatch.py", line 25, in __call_
return self.callable(*self.args, **self.kwargs)
File "<string>", line 1, in <lambda>
File "E:\Splunk\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\lib\decorators.py", line 38, in rundecs
return fn(*a, **kw)
File "<string>", line 1, in <lambda>
File "E:\Splunk\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\lib\decorators.py", line 118, in check
return fn(self, *a, **kw)
File "<string>", line 1, in <lambda>
File "E:\Splunk\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\lib\decorators.py", line 167, in validate_ip
return fn(self, *a, **kw)
File "<string>", line 1, in <lambda>
File "E:\Splunk\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\lib\decorators.py", line 246, in preform_sso_check
update_session_user(sessionKey, remote_user)
File "E:\Splunk\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\lib\decorators.py", line 189, in update_session_user
en = splunk.entity.getEntity('authentication/users', user, sessionKey=sessionKey)
File "E:\Splunk\Python-2.7\Lib\site-packages\splunk\entity.py", line 249, in getEntity
serverResponse, serverContent = rest.simpleRequest(uri, getargs=kwargs, sessionKey=sessionKey, raiseAllErrors=True)
File "E:\Splunk\Python-2.7\Lib\site-packages\splunk\rest_init_.py", line 567, in simpleRequest
raise splunk.RESTException, (serverResponse.status, serverResponse.messages)
RESTException: [HTTP 503] General server error
... View more