Deployment Architecture

Splunk search head issues running on a VM (memory leak?)

bruceclarke
Contributor

All,

I have a search head running on a VM that reads from two search indexers (also on VMs). I've been having issues with the search head's virtual machine.

About once every week or two, we can no longer log onto the head's VM and the Splunk front end won't allow logins either (the front end gives us a ResponseNotReady error). When trying to log onto the VM, we get an error saying:

"Remote Desktop cannot verify the identity of the remote computer because there is a time or date difference between your computer and the remote computer. Make sure your computer's clock is set to the correct time, and then try connecting again. If the problem occurs again, contact your network administrator or the owner of the remote computer."

After we reboot the machine, everything goes back to normal.

We initially thought that there was an issue with the time on our VM, but further inspection showed that the time was right and the machine was also failing to do domain authentication in general. Our best guess now is that there is a memory leak issue with the Splunk service (though this is a little odd, since the indexers run just fine).

Has anyone seen this issue with search heads on VMs before? Like I said, we don't see the issue with the indexers, which are also on VMs. Are there any limits I can set in limits.conf to prevent Splunk from using too much memory?

Thanks!

Tags (3)
1 Solution

Jason
Motivator

This error is likely due to Splunk being unable to talk to itself. It's a known issue with Windows, see:

http://docs.splunk.com/Documentation/Splunk/latest/ReleaseNotes/Workaroundfornetworkaccessibilityiss...

This answer has the most detailed information, in terms of users' experience.

http://answers.splunk.com/answers/68368/splunk-web-throws-responsenotready-error

View solution in original post

Jason
Motivator

This error is likely due to Splunk being unable to talk to itself. It's a known issue with Windows, see:

http://docs.splunk.com/Documentation/Splunk/latest/ReleaseNotes/Workaroundfornetworkaccessibilityiss...

This answer has the most detailed information, in terms of users' experience.

http://answers.splunk.com/answers/68368/splunk-web-throws-responsenotready-error

View solution in original post

bruceclarke
Contributor

Hey @dart, if you want to convert your comment to an answer, you definitely solved it for me! Thanks!

0 Karma

bruceclarke
Contributor

watsm10 - I was able to reduce the occurrences of this issue by throttling the number of concurrent searches in the limits.conf file. That said, I still have seen the issue occasionally (maybe once a month). It looks like dart has the solution I've been looking for.

dart - Thanks for the link. I'll try it out and see if that's the issue. I see the error in the splunkd.log file, so I'm pretty confident this is the issue.

0 Karma

dart
Splunk Employee
Splunk Employee

watsm10
Contributor

Hi Bruce,

I'm currently having the same issues. I'm not sure what the issue is with our VM's either. Did you get anywhere with yours?

0 Karma
Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!