extremely slow Search head on windows 2008 R2 standard

[aandrew]

Hi, is anyone out there having a Slow search and missed alerts on Search head.
we have installed search head on 64 bit - Windows 2008 R2 Standard with Intel Xeon R CPU E7-8837 @ 2.67 GHz with 4 Processors and 32 GB of Mem.
after running Splunk for 3 days the server started to slow down drastically. we are not able to RDP to the box it's self..
when we look at the IO , CPU , and Mem usage is very low yet the splunk will miss all alerts and unable to search.
currently we are restarting the server to resolve this issue. I would like to know if any one have a fix for this...

[aandrew]

Thank you so much for all of your help... yes i have created a support case and i will post the findings...

[sowings]

Since the collocation of deployment server and deployment client is not the issue with your search head, I'd recommend opening a support case. The S.o.S. app may help you identify some resource contention, but ultimately, you'll want to identify a root cause.

[aandrew]

I have checked our deployment and we do have a standalone deployment server and it's running on VM...
our clients are less then 40 at the moment..

[sowings]

Since you've indicated that the search head is simultaneously a deployment server and a deployment client of itself, you'll want to separate the two functions. It's probably easiest to move the deployment server to a separate host. This system doesn't have to have a lot of resources, unless your deployment environment is very large (> 3000 hosts). You may see some alleviation of your symptoms by disabling (renaming with a different file extension) the deploymentclient.conf on the search head, and restarting so that the deployment client feature is thereby disabled.

Once you're able to relocate the deployment server functionality (move etc/deployment-apps and etc/system/local/serverclass.conf) to a new system, you can once again make the search head a deployment client.

[bmacias84]

If you are running your Search Head as a Deployment Server how many Deployment Clients do you have and what is your check-in interval.

[aandrew]

yes we are using Deployment server. is there document on how to split Server from client?

[sowings]

Are you using the Deployment Server? I've observed the search head UI behaving in the way you describe when the deployment server was acting as a deployment client as well. The solution in that instance is to separate out the deployment server from any deployment clients.

[alacercogitatus]

If you are positive that it's Splunk, you may want to get the S.O.S. (splunk on splunk) app and see if there are any problems with the indexing queues and whatnot, but the box seems beefy enough to run a standard install just fine.

Do you have an enterprise license?

http://splunk-base.splunk.com/apps/29008/sos-splunk-on-splunk

[aandrew]

We have S.O.S and no issue there but only time we can get to it after the restart... 😞