All Apps and Add-ons

Error Configuring Hadoop Connect App--Failed to find any Kerberos tgt(BOUNTY OFFERED!)

nekbote
Path Finder

Hi Team,

I am trying to configure the splunk hadoop connect app to connect to kerberos secured cloudera hadoop and i am running into issues.

My Kerberos hadoop admin created the keytab file and shared it with me. I have copied the keytab file /home/myuser/abc.keytab.

In the splunk hadoop connect app ui i have added the kerberos principal with Principal Name -->abc@xyz.REFINERY.QA and Path as /home/myuser/abc.keytab and it gets saved without error and shows up in kerberos principal section in splunk hadoop connect app ui.

From the shell on the search head am able to run below commands successfully

kinit -k -t /home/myuser/abc.keytab  abc@xyz.REFINERY.QA ---runs fine i do not get error nor prompt to enter password
klist command works successfully ....klist command output as below

Ticket cache: FILE:/tmp/krb5cc_30074
Default principal: abc@xyz.REFINERY.QA

Valid starting     Expires            Service principal
06/22/17 16:11:42  06/23/17 16:11:42  krbtgt/xyz.REFINERY.QA@xyz.REFINERY.QA
    renew until 06/29/17 16:11:38

I have configured the krb5.conf file on my search head under /etc/krb5.conf to have below values

krb5.conf   
[libdefaults]
default_realm = xyz.REFINERY.QA
dns_lookup_kdc = false
dns_lookup_realm = false
ticket_lifetime = 86400
renew_lifetime = 604800
forwardable = true
default_tgs_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 des3-cbc-sha1 arcfour-hmac des-hmac-sha1 des-cbc-md5
default_tkt_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 des3-cbc-sha1 arcfour-hmac des-hmac-sha1 des-cbc-md5
permitted_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 des3-cbc-sha1 arcfour-hmac des-hmac-sha1 des-cbc-md5
udp_preference_limit = 1
[realms]
xyz.REFINERY.QA = {
kdc = QAhost01
admin_server = QAhost01
}

Our Hadoop is a clustered and below are the details

Splunk Version : 6.5.2
Cloudera Enterprise 5.10.1 (hadoop-2.6.0)
Kerberos Secured

HDFS site xml value which i am using in the splunk hadoop connect UI

<configuration>
   <property>
     <name>dfs.nameservices</name>
     <value>namsvc10</value>
   </property>
   <property>
     <name>dfs.ha.namenodes.namsvc10</name>
     <value>namenode1,namenode2</value>
   </property>
   <property>
     <name>dfs.namenode.rpc-address.namsvc10.namenode1</name>
     <value>QAhost01:8020</value>
   </property>
   <property>
     <name>dfs.namenode.rpc-address.namsvc10.namenode2</name>
     <value>QAhost02:8020</value>
   </property>
   <property>
     <name>dfs.namenode.http-address.namsvc10.namenode1</name>
     <value>QAhost01:50070</value>
   </property>
   <property>
     <name>dfs.namenode.http-address.namsvc10.namenode2</name>
     <value>QAhost02:50070</value>
   </property>
   <property>
     <name>dfs.client.failover.proxy.provider.namsvc10</name>
     <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
   </property>
 </configuration>

am able to generate the ticket from the shell(using kinit and klist) and able to run above commands but when i fill in all the config details in splunk hadoop connect app ui and save i get below error

Unable to connect to Hadoop cluster 'hdfs://namsvc10/' with principal 'abc@xyz.REFINERY.QA': Failed to run Hadoop CLI job command '-ls' with options 'hdfs://namsvc10/': ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "mylocalhost"; destination host is: "QAhost02:8020; 

Any idea what am i missing, why does it work from shell and NOT through the splunk hadoop connect app ui.

0 Karma

nekbote
Path Finder

Also from the shell when i am running the command, I am getting below Error.

./bin/hadoop fs -ls hdfs://QAhost02:8020

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
WARN security.UserGroupInformation: PriviledgedActionException as:abc (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
WARN security.UserGroupInformation: PriviledgedActionException as:abc (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "mylocalhost"; destination host is: "QAhost02":8020;  

Also Our Hadoop admin suggested that we need to have unix local account on the search head with the same name as the Kerberos user. Say our Kerberos user is "abc" , on splunk search head we need to be logged in as "abc" and then run the above command for it to work. We implemented this out but still continue to get this error. My Hadoop adminstrator now suspects my local dev host has to be part of their KDC trusted list as my splunk dev box is outside of hadoop cluster network we are getting this error.

0 Karma

rdagan_splunk
Splunk Employee
Splunk Employee

Since it looks like it will require some debugging .. have you opened a support ticket with splunk support?

0 Karma

nekbote
Path Finder

Hi rdagan,

No I haven't yet. I am waiting to validate the below scenario and then raise a ticket or reach out to our splunk support contact.

Right now from the splunk search head CLI , we are not able to connect to hadoop cluster though we have the valid ticket. We ran this command as admin, we are receiving invalid tgt error.

./bin/hadoop fs -ls hdfs://servername:8020

Our Hadoop admin suggested that we need to have unix local account on the search head with the same name as the Kerberos user. Say our Kerberos user is "abc" , on splunk search head we need to be logged in as "abc" and then run the above command for it to work. I am waiting on getting this done and then plan to log the ticket.

Do you know if the user needs to be same as the Kerberos user, I don't think this would matter but thought we should be able to connect from CLI to hadoop then we know there is something missing on the hadoop connect config side.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...