topic DFS manager: Spark node failed to initialize in Splunk Enterprise

DFS manager: Spark node failed to initialize

rabindrakumarpa — Mon, 15 Jun 2020 19:26:40 GMT

Hi All,

Post deploying DFS manager on search Head(Master), when I am trying to start splunk, I am seeing below error:

“The Spark node failed to initialize" after debug message “Building configuration information”.

Also, DFS manager(Application Page) says "Unable to connect Spark master", although Splunk searchhead is working fine.

Checked and ports are open 8005-8010, Changed Java_home in splunk_dfs_manager.sh and spark_app.conf as it was complaining about missing Java_home.

Attaching snippet of Splunkd log.

Please advise.

Re: DFS manager: Spark node failed to initialize

burwell — Sun, 14 Jun 2020 01:55:23 GMT

Have you tried the newest DFS manager 1.2.1? That might help.

Re: DFS manager: Spark node failed to initialize

rabindrakumarpa — Mon, 15 Jun 2020 06:01:10 GMT

I just reworked with DFS manager 1.2.1.. Although the message is little meaningful but yet problem persists..

"Failed to get spark home path from app configuration."

I checked spark_app.conf, it seems to be proper.. However, server.conf is having sparkhome as blank... I attempted to update that as well and tried but issue remained the same.

spark_home = /opt/splunk/etc/apps/splunk_dfs_manager/vendor/spark-2.3.3-bin-hadoop2.7

Thanks In advance.

Re: DFS manager: Spark node failed to initialize

rabindrakumarpa — Mon, 15 Jun 2020 06:01:46 GMT

Re: DFS manager: Spark node failed to initialize

burwell — Mon, 15 Jun 2020 16:41:45 GMT

I don't have spark_home set and don't have an issue.

I think you should file a case with Splunk support. DFS config with the splunk_dfs_manager is definitely finicky.

Re: DFS manager: Spark node failed to initialize

rabindrakumarpa — Tue, 16 Jun 2020 05:28:18 GMT

Thank You! Sure we will raise case with splunk..

Do we have any recommended configuration strategy to enable data Fabric search without DFS manager .. ? Any doc/Link in that line will be quite helpful..

We are following..

https://docs.splunk.com/Documentation/DFS/1.1.1/DFS/InstallationChecklist

Thanks in advance..

Re: DFS manager: Spark node failed to initialize

burwell — Tue, 16 Jun 2020 15:35:11 GMT

Hi. I was a beta tester of Splunk 7.3.1 and we "rolled our own" as the DFS manager was not available.

Starting with 8.0 we used the DFS manager.

The documentation says they don't support rolling your own.

You have to install Spark 2.3.3 and Java yourself on your Spark nodes and manage the Spark configurations.