Excellent insights.  To add to original question,  I am getting ready to turn on SmartStore on my IDX Cluster as part of a SplunkCloud Migration.  What could go wrong and is there any backing out if problems develop? We are using Splunk PS and a migration app. ... View more

I got this error as well because the java jre path was removed or changed. The Java on my standalone dev Splunk server got updated by our SysAdmins and I was getting the "Can not communicate with task server, check your settings" error. It had been working just fine. I went to settings and reviewed them. I verified the java path with what was on the server and sure enough the old java path was gone. I updated the java path to the new java path and saved. It attempted the restart of the task server but failed. I restarted splunk and everything came up working like normal. ... View more

Another way this error will occur is if the file ownership or permissions on your Splunk directories are changed. I had an admin inadvertently move ownership of /splunk on my servers and it immediately started throwing these 500 errors to my splunk users. A simple 'ls -lar' or splunk status will confirm. Once ownership was restored the 500 errors went away. Some search and alerting functions did require a restart of splunk to fix. ... View more