500 internal server Error and Windows winsock error 10055 (splunkd.log) after upgrading to 5.0.1

[Masa]

Windows winsock error 10055 after upgrading to 5.0.1

We have Windows 2k8 servers and 2k3 server for three search heads and two indexers. We upgraded from 4.3.4 to 5.0.1. Then, we started to have issues of accessing to Splunkweb with "500 Internal server error" a few times a week. Then, we realized that we cannot RDP to the server, either. We had to reboot the server.

It seems like there is no issue on the indexers because I could login to them and run searches directly to the indexers.

As far as I know, we haven't added any apps. But, I'm sure there are more scheduled searches were created by users. I'm not sure how to troubleshoot this.

[Masa · Accepted solution]

We know v5.0.1 uses more resources. Probably that's the reason you're reaching the limit of Windows Socket buffer.

According to Microsoft Website, the error means;

Winsock Error 10055:
No buffer space available. An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.

If that's the case, solution would be;

Apply a hotfix from Microsoft
( http://support.microsoft.com/kb/2577795 )

Or,

Change the limit in the registry directly
( http://support.microsoft.com/kb/Q196271 )

So far, we haven't seen the error after applying the hotfix.

You can also find more details regarding this issue in this documentation topic.