Splunk Hadoop Connect: How to troubleshoot error "failure to login...Unable to connect to hadoop cluster"?

[kausar]

Hi,

I am using the plunk Hadoop Connect App to add a cluster and connect to a remote hdfs. After filling the form, when I hit save, I get the following error (in splunk/python.log OR splunk/HadoopConnect.log). I have checked the keytab file - it works fine when doing, kinit -k -t principal.keytab and there is no ACL issue - hadoop cmds are working fine locally from box.

2016-08-15 17:51:39,179 ERROR Failed to run hadoop cli, cmd=-ls, options=('hdfs://<namenode>:8020/',), error=ls: failure to login
    2016-08-15 17:51:39,180 ERROR Failed to validate hadoop connection
    Traceback (most recent call last):
      File "/opt/splunk/etc/apps/HadoopConnect/bin/hadooputils.py", line 187, in validateConnectionToHadoop
        hj.wait(True)
      File "/opt/splunk/etc/apps/HadoopConnect/bin/hadooputils.py", line 422, in wait
        raise HcException(HCERR0001, {'cmd':self.fscmd, 'options':','.join(self.fsopts), 'error':self.getStderr()})
    HcException: {"id": "HCERR0001", "options": "hdfs://<namenode>:8020/", "error": "ls: failure to login", "cmd": "-ls", "message": "Failed to run hadoop CLI Job"}
    2016-08-15 17:51:39,181 ERROR Failed to handleEdit
    Traceback (most recent call last):
      File "/opt/splunk/etc/apps/HadoopConnect/bin/clusters_handler.py", line 96, in handleEdit
        self.handleHdfsCreateOrEdit(cluster, confInfo)
      File "/opt/splunk/etc/apps/HadoopConnect/bin/clusters_handler.py", line 229, in handleHdfsCreateOrEdit
        hu.validateConnectionToHadoop(self.getSessionKey(), principal, 'hdfs://'+self.callerArgs.id+'/')
      File "/opt/splunk/etc/apps/HadoopConnect/bin/hadooputils.py", line 199, in validateConnectionToHadoop
        raise HcException(HCERR0015, {'uri':uri, 'principal':principal, 'error':msg})
    HcException: {"id": "HCERR0015", "uri": "hdfs://<namenode>:8020/", "principal": "user@abc.xyz", "error": "Failed to run Hadoop CLI job command '-ls' with options 'hdfs://<namenode>:8020/': ls: failure to login.", "message": "Unable to connect to hadoop cluster"}

[kausar · Accepted solution]

We were missing the "Kerberos Principals to Short Names Mapping" field in the configure form (initially left blank). It worked when copied the output of the following command (along with spaces).

$ hdfs getconf -confKey hadoop.security.auth_to_local

Splunk reference:
http://docs.splunk.com/Documentation/HadoopConnect/1.2.3/DeployHadoopConnect/Configuretheapp

[newbie2tech]

Hi @kausar,

I am getting same error inspite of listing down the "Kerberos Principals to Short Names Mapping" value as well. any thoughts what else could be missing