Hi All, I have integrated splunk with hadoop using hadoop connect app in my search head instance [non-clustered search head] and the data export to hadoop is working fine. As part of this one of the steps is to copy the hadoop related details in to krb5.conf, which i did in the default file /etc/krb5.conf. Now looks like my changes overwrote the previous existing contents due to which some of the power broker roles on the machine are no longer working and the splunk admin is no longer able to sudo due to which some of his monitoring scripts are failing as well. We got to know that power broker login also makes use of the krb5.conf which comes built in with machine @ my organization. On checking with our Hadoop admin we got to know that appending of the Hadoop related configs to default krb5.conf isn't an option [as cross relams are not supported currently] and we need to have 2 separate config files and specify the paths of these 2 files by setting "KRB5_CONFIG" environment variable. Example value for multiple file specification --> KRB5_CONFIG =/etc/default_krb5.conf:/etc/hadoop_krb5.conf [reference link https://web.mit.edu/kerberos/krb5-devel/doc/admin/env_variables.html ] Questions To point Hadoop Connect app to pick up the custom file configuration, can you please help share the list of files and the property name that needs to be changed. Would be great if you can share some suggestions. Thanks! Splunk Version : 6.5.2 Cloudera Enterprise 5.10.1 (hadoop-2.6.0) Kerberos Secured Hadoop Connect App : 1.2.5 ... View more

Hi Team, I am trying to configure the splunk hadoop connect app to connect to kerberos secured cloudera hadoop and i am running into issues. My Kerberos hadoop admin created the keytab file and shared it with me. I have copied the keytab file /home/myuser/abc.keytab. In the splunk hadoop connect app ui i have added the kerberos principal with Principal Name -->[email protected] and Path as /home/myuser/abc.keytab and it gets saved without error and shows up in kerberos principal section in splunk hadoop connect app ui. From the shell on the search head am able to run below commands successfully kinit -k -t /home/myuser/abc.keytab [email protected] ---runs fine i do not get error nor prompt to enter password klist command works successfully ....klist command output as below Ticket cache: FILE:/tmp/krb5cc_30074 Default principal: [email protected] Valid starting Expires Service principal 06/22/17 16:11:42 06/23/17 16:11:42 krbtgt/[email protected] renew until 06/29/17 16:11:38 I have configured the krb5.conf file on my search head under /etc/krb5.conf to have below values krb5.conf [libdefaults] default_realm = xyz.REFINERY.QA dns_lookup_kdc = false dns_lookup_realm = false ticket_lifetime = 86400 renew_lifetime = 604800 forwardable = true default_tgs_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 des3-cbc-sha1 arcfour-hmac des-hmac-sha1 des-cbc-md5 default_tkt_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 des3-cbc-sha1 arcfour-hmac des-hmac-sha1 des-cbc-md5 permitted_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 des3-cbc-sha1 arcfour-hmac des-hmac-sha1 des-cbc-md5 udp_preference_limit = 1 [realms] xyz.REFINERY.QA = { kdc = QAhost01 admin_server = QAhost01 } Our Hadoop is a clustered and below are the details Splunk Version : 6.5.2 Cloudera Enterprise 5.10.1 (hadoop-2.6.0) Kerberos Secured HDFS site xml value which i am using in the splunk hadoop connect UI <configuration> <property> <name>dfs.nameservices</name> <value>namsvc10</value> </property> <property> <name>dfs.ha.namenodes.namsvc10</name> <value>namenode1,namenode2</value> </property> <property> <name>dfs.namenode.rpc-address.namsvc10.namenode1</name> <value>QAhost01:8020</value> </property> <property> <name>dfs.namenode.rpc-address.namsvc10.namenode2</name> <value>QAhost02:8020</value> </property> <property> <name>dfs.namenode.http-address.namsvc10.namenode1</name> <value>QAhost01:50070</value> </property> <property> <name>dfs.namenode.http-address.namsvc10.namenode2</name> <value>QAhost02:50070</value> </property> <property> <name>dfs.client.failover.proxy.provider.namsvc10</name> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> </property> </configuration> am able to generate the ticket from the shell(using kinit and klist) and able to run above commands but when i fill in all the config details in splunk hadoop connect app ui and save i get below error Unable to connect to Hadoop cluster 'hdfs://namsvc10/' with principal '[email protected]': Failed to run Hadoop CLI job command '-ls' with options 'hdfs://namsvc10/': ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "mylocalhost"; destination host is: "QAhost02:8020; Any idea what am i missing, why does it work from shell and NOT through the splunk hadoop connect app ui. ... View more