Archive

HUNK with Resource Manager High Availability: Is Hunk able to use two resource manager service names and use active service?

Path Finder

Hi all,

The problem is in YARN Resource Manager service, which resides on namenodes and it is also configured as HA service, but there is no HA service name in Resouce Manager HA configuration, so I must add active namenode name in hunk configuration.

If Active namenode fails and secondary namenode becomes active, hunk is able to read hdfs information correctly, but since there is only one Resource Manager hostname given in Hunk configuration, it will not be able to use YARN application for data management processes.

YARN Resource Manager service becomes single point of failure if HA is not configured for this service and active namenode goes down.
I used documentation below to configure Resource Manager HA:
http://hadoop.apache.org/docs/r2.5.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html

Is Hunk able to use two Resource manager service names and use active service?

Regards,
Dmitry

Tags (3)
0 Karma
1 Solution

Splunk Employee
Splunk Employee

To enable Hunk to work with Yarn HA add the appropriate flags to the Provider.
You can find all these values under resource manager ip:8088/conf

In the Provider UI -> all the way at the bottom -> Add the below flags. Note that these flags should be identical you your configurations (just add vix at the beginning of each flag)

These are example flags you will need to add to the Provider.
After you are done with adding these flags you can validate your input in the file /hunk/etc/apps/search/local/indexes.conf

vix.yarn.resourcemanager.ha.rm-ids = rm1,rm2
vix.yarn.resourcemanager.address.rm1 = host1:23140
vix.yarn.resourcemanager.scheduler.address.rm1 = host1:23130
vix.yarn.resourcemanager.address.rm2 = host2:23140
vix.yarn.resourcemanager.scheduler.address.rm2 = host2:23130
vix.yarn.client.failover-proxy-provider = org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider

View solution in original post

Splunk Employee
Splunk Employee

To enable Hunk to work with Yarn HA add the appropriate flags to the Provider.
You can find all these values under resource manager ip:8088/conf

In the Provider UI -> all the way at the bottom -> Add the below flags. Note that these flags should be identical you your configurations (just add vix at the beginning of each flag)

These are example flags you will need to add to the Provider.
After you are done with adding these flags you can validate your input in the file /hunk/etc/apps/search/local/indexes.conf

vix.yarn.resourcemanager.ha.rm-ids = rm1,rm2
vix.yarn.resourcemanager.address.rm1 = host1:23140
vix.yarn.resourcemanager.scheduler.address.rm1 = host1:23130
vix.yarn.resourcemanager.address.rm2 = host2:23140
vix.yarn.resourcemanager.scheduler.address.rm2 = host2:23130
vix.yarn.client.failover-proxy-provider = org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider

View solution in original post