HadoopConnect unable to add HDFS cluster

[jon_doll]

Hello - we are in the process of replacing our HadoopConnect server environment (1 SH, 4 Indexers, 1 cluster master running Splunk Enterprise 7.0.6 on Linux 6) with new Linux 7 hardware and running Splunk 7.3.3.

When attempting to set up the HDFS cluster on the new box through the UI, we get "Failed to get entities' object ''.

If I try to configure through the clusters.conf file, it doesn't show up in the UI at all, and when trying to 'explore' the HDFS, I get Error in 'hdfs' command: Failed to get entities' object 'admin/clusters'.

When I run the command directly through the command line it seems to work.

bash-4.2$ $HADOOP_HOME/bin/hadoop fs -ls hdfs://xr1ph010:8020/

Found 15 items

drwxrwxrwx   - yarn   hadoop                  0 2021-03-22 10:29 hdfs://xr1ph010:8020/app-logs

drwxr-xr-x+  - hdfs   hdfs                    0 2020-03-11 08:18 hdfs://xr1ph010:8020/apps

drwxr-xr-x   - yarn   hadoop                  0 2016-07-29 20:09 hdfs://xr1ph010:8020/ats

drwxrwxrwx   - hdfs   p-l-hdp-birs-x          0 2021-03-18 16:07 hdfs://xr1ph010:8020/benchmarks

drwxr-xr-x+  - hdfs   hdfs                    0 2021-03-04 11:25 hdfs://xr1ph010:8020/data

drwxrwxr-x   - hdfs   hdfs                    0 2017-02-23 11:02 hdfs://xr1ph010:8020/datascience

drwxrwxr-x   - hdfs   hadoop                  0 2015-09-19 00:12 hdfs://xr1ph010:8020/hdp

drwxr-xr-x+  - hdfs   hdfs                    0 2015-01-29 21:43 hdfs://xr1ph010:8020/lost+found

drwxr-xr-x   - mapred hdfs                    0 2013-11-07 09:59 hdfs://xr1ph010:8020/mapred

drwxr-xr-x   - hive   hdfs                    0 2020-01-30 09:46 hdfs://xr1ph010:8020/mnt

drwxrwxrwx   - hdfs   hdfs                    0 2013-11-07 09:59 hdfs://xr1ph010:8020/mr-history

drwxrwxr-x   - hdfs   hdfs                    0 2016-11-13 01:10 hdfs://xr1ph010:8020/ranger

drwxr-xr-x+  - hdfs   hdfs                    0 2021-01-11 15:47 hdfs://xr1ph010:8020/system

drwxrwxrwx   - hdfs   hdfs                    0 2021-03-22 09:10 hdfs://xr1ph010:8020/tmp

drwxr-xr-x   - hdfs   hdfs                    0 2021-03-11 10:56 hdfs://xr1ph010:8020/user

The two main errors I see in the HadoopConnect log are HCERR2002 and HCERR0501

2021-04-06 08:55:07,864 ERROR hdfs_search_command.py [<module>] [341] - sid=1617713707.4, {"message": "Missing required argument", "id": "HCERR0501", "argument": "uri"}
Traceback: Traceback (most recent call last):
File "/opt/splunk/etc/apps/HadoopConnect/bin/hdfs_search_command.py", line 330, in <module>
hdfs.main()
File "/opt/splunk/etc/apps/HadoopConnect/bin/hdfs_search_command.py", line 323, in main
self._main_impl()
File "/opt/splunk/etc/apps/HadoopConnect/bin/hdfs_search_command.py", line 290, in _main_impl
raise HcException(HCERR0501, {'argument':'uri'})
HcException: {"message": "Missing required argument", "id": "HCERR0501", "argument": "uri"}

2021-04-06 08:51:13,583 ERROR hdfs_search_command.py [<module>] [341] - sid=1617713473.3, {"message": "Failed to get entities object", "id": "HCERR2002", "uri": "", "entity_path": "admin/clusters", "search": "", "error": "Unexpected error \"<class 'errors.HcException'>\" from python handler: \"{\"search\": \"\", \"entity_path\": \"\", \"error\": \"'NoneType' object has no attribute 'startswith'\", \"id\": \"HCERR2002\", \"message\": \"Failed to get entities object\", \"uri\": \"/servicesNS/jdoll1/HadoopConnect/configs/conf-clusters\"}\". See splunkd.log for more details."}
Traceback: Traceback (most recent call last):
File "/opt/splunk/etc/apps/HadoopConnect/bin/hdfs_search_command.py", line 330, in <module>
hdfs.main()
File "/opt/splunk/etc/apps/HadoopConnect/bin/hdfs_search_command.py", line 323, in main
self._main_impl()
File "/opt/splunk/etc/apps/HadoopConnect/bin/hdfs_search_command.py", line 293, in _main_impl
self._validateURI(k)
File "/opt/splunk/etc/apps/HadoopConnect/bin/hdfs_search_command.py", line 71, in _validateURI
raise HcException(HCERR2002, {'entity_path':'admin/clusters', 'search':'', 'uri':'', 'error':msg})
HcException: {"message": "Failed to get entities object", "id": "HCERR2002", "uri": "", "entity_path": "admin/clusters", "search": "", "error": "Unexpected error \"<class 'errors.HcException'>\" from python handler: \"{\"search\": \"\", \"entity_path\": \"\", \"error\": \"'NoneType' object has no attribute 'startswith'\", \"id\": \"HCERR2002\", \"message\": \"Failed to get entities object\", \"uri\": \"/servicesNS/jdoll1/HadoopConnect/configs/conf-clusters\"}\". See splunkd.log for more details."}