Does Hunk App for Hbase support a Kerberized configuration?

[eangeles]

Been trying to configure 'Hunk App for HBase' with a Kerberized cluster and not having much luck. I was able to configure it with a non-secure hbase no problem. The HBase version is 1.1.2.2.5.0.0-1245 and Splunk version is 6.5.1.

Is the application compatible with Kerberos? I can't seem to decipher anything from the logs - seems like its connecting?

02-03-2017 17:05:47.349 ERROR ERP.hbase -  hconnection-0x9d14a90x0, quorum=master1.hadoop.companyABC.com:2181,slave1.hadoop.companyABC.com:2181,master2.hadoop.companyABC.com:2181, baseZNode=/hbase-secure Received ZooKeeper Event, type=None, state=SyncConnected, path=null
02-03-2017 17:05:47.352 ERROR ERP.hbase -  hconnection-0x9d14a9-0x35911e84197da93 connected
02-03-2017 17:05:47.354 ERROR ERP.hbase -  Reading reply sessionid:0x35911e84197da93, packet:: clientPath:null serverPath:null finished:false header:: 1,3  replyHeader:: 1,1430239131173,0  request:: '/hbase-secure/hbaseid,F  response:: s{601295421767,1430236207061,1450390354271,1485257583338,56,0,0,0,67,0,601295421767} 
02-03-2017 17:05:47.359 ERROR ERP.hbase -  Reading reply sessionid:0x35911e84197da93, packet:: clientPath:null serverPath:null finished:false header:: 2,4  replyHeader:: 2,1430239131173,0  request:: '/hbase-secure/hbaseid,F  response:: #ffffffff000146d61737465723a3136303030ffffffc31bffffffea681fffffff2696150425546a2432633833323230312d643239652d346133392d616635662d323037313831306639323736,s{601295421767,1430236207061,1450390354271,1485257583338,56,0,0,0,67,0,601295421767} 
02-03-2017 17:05:47.493 ERROR ERP.hbase -  Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@22be9f8f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=null
02-03-2017 17:05:47.498 ERROR ERP.hbase -  Getting table 'my_reference_table' and column family '*'
02-03-2017 17:05:47.513 ERROR ERP.hbase -  Reading reply sessionid:0x35911e84197da93, packet:: clientPath:null serverPath:null finished:false header:: 3,3  replyHeader:: 3,1430239131173,0  request:: '/hbase-secure,F  response:: s{601295421598,601295421598,1450390322736,1450390322736,0,240,0,0,0,20,1430236207148} 
02-03-2017 17:05:47.517 ERROR ERP.hbase -  Reading reply sessionid:0x35911e84197da93, packet:: clientPath:null serverPath:null finished:false header:: 4,4  replyHeader:: 4,1430239131173,0  request:: '/hbase-secure/master,F  response:: #ffffffff000146d61737465723a31363030304affffff9232ffffffcb303137ffffffda50425546a28a1c6d6173746572312e6861646f6f702e74736c2e74656c75732e636f6d10ffffff807d18ffffff99ffffffe7ffffffffffffff81ffffff9d2b10018ffffff8a7d,s{1430236207048,1430236207048,1485257581917,1485257581917,0,0,0,241243744271518884,76,0,1430236207048} 
02-03-2017 17:05:47.689 ERROR ERP.hbase -  Use SIMPLE authentication for service MasterService, sasl=false
02-03-2017 17:05:47.706 ERROR ERP.hbase -  Connecting to master1.hadoop.companyABC.com/192.168.1.10:16000
02-03-2017 17:05:47.917 ERROR ERP.hbase -  Reading reply sessionid:0x35911e84197da93, packet:: clientPath:null serverPath:null finished:false header:: 5,3  replyHeader:: 5,1430239131173,0  request:: '/hbase-secure,F  response:: s{601295421598,601295421598,1450390322736,1450390322736,0,240,0,0,0,20,1430236207148} 
02-03-2017 17:05:47.919 ERROR ERP.hbase -  Reading reply sessionid:0x35911e84197da93, packet:: clientPath:null serverPath:null finished:false header:: 6,4  replyHeader:: 6,1430239131173,0  request:: '/hbase-secure/master,F  response:: 

[eangeles]

I've put in the settings you mentioned in the comments and it seems to be still unable to connect. I've tried it both ways via the keytab as well as via the ticket cache.

Below is the output of the logs:

02-16-2017 16:56:07.043 ERROR ERP.hbase -  Reading reply sessionid:0x158d1da9640bb9a, packet:: clientPath:null serverPath:null finished:false header:: 4,4  replyHeader:: 4,1430241685179,0  request:: '/hbase-secure/hbaseid,F  response:: #ffffffff000146d61737465723a3136303030ffffffc31bffffffea681fffffff2696150425546a2432633833323230312d643239652d346133392d616635662d323037313831306639323736,s{601295421767,1430236207061,1450390354271,1485257583338,56,0,0,0,67,0,601295421767} 
02-16-2017 16:56:07.167 ERROR ERP.hbase -  Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@344113f8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=null
02-16-2017 16:56:07.173 ERROR ERP.hbase -  Getting table 'my_reference_table' and column family '*'
02-16-2017 16:56:07.188 ERROR ERP.hbase -  Reading reply sessionid:0x158d1da9640bb9a, packet:: clientPath:null serverPath:null finished:false header:: 5,3  replyHeader:: 5,1430241685179,0  request:: '/hbase-secure,F  response:: s{601295421598,601295421598,1450390322736,1450390322736,0,240,0,0,0,20,1430236207148} 
02-16-2017 16:56:07.193 ERROR ERP.hbase -  Reading reply sessionid:0x158d1da9640bb9a, packet:: clientPath:null serverPath:null finished:false header:: 6,4  replyHeader:: 6,1430241685179,0  request:: '/hbase-secure/master,F  response:: #ffffffff000146d61737465723a31363030304affffff9232ffffffcb303137ffffffda50425546a28a1c6d6173746572312e6861646f6f702e74736c2e74656c75732e636f6d10ffffff807d18ffffff99ffffffe7ffffffffffffff81ffffff9d2b10018ffffff8a7d,s{1430236207048,1430236207048,1485257581917,1485257581917,0,0,0,241243744271518884,76,0,1430236207048} 
02-16-2017 16:56:07.359 ERROR ERP.hbase -  Use SIMPLE authentication for service MasterService, sasl=false
02-16-2017 16:56:07.375 ERROR ERP.hbase -  Connecting to master1.hadoop.tsl.companyABC.com/192.168.1.10:16000
02-16-2017 16:56:07.586 ERROR ERP.hbase -  Reading reply sessionid:0x158d1da9640bb9a, packet:: clientPath:null serverPath:null finished:false header:: 7,3  replyHeader:: 7,1430241685179,0  request:: '/hbase-secure,F  response:: s{601295421598,601295421598,1450390322736,1450390322736,0,240,0,0,0,20,1430236207148} 
02-16-2017 16:56:07.589 ERROR ERP.hbase -  Reading reply sessionid:0x158d1da9640bb9a, packet:: clientPath:null serverPath:null finished:false header:: 8,4  replyHeader:: 8,1430241685179,0  request:: '/hbase-secure/master,F  response:: #ffffffff000146d61737465723a31363030304affffff9232ffffffcb303137ffffffda50425546a28a1c6d6173746572312e6861646f6f702e74736c2e74656c75732e636f6d10ffffff807d18ffffff99ffffffe7ffffffffffffff81ffffff9d2b10018ffffff8a7d,s{1430236207048,1430236207048,1485257581917,1485257581917,0,0,0,241243744271518884,76,0,1430236207048} 
02-16-2017 16:56:07.893 ERROR ERP.hbase -  Reading reply sessionid:0x158d1da9640bb9a, packet:: clientPath:null serverPath:null finished:false header:: 9,3  replyHeader:: 9,1430241685179,0  request:: '/hbase-secure,F  response:: s{601295421598,601295421598,1450390322736,1450390322736,0,240,0,0,0,20,1430236207148} 
02-16-2017 16:56:07.896 ERROR ERP.hbase -  Reading reply sessionid:0x158d1da9640bb9a, packet:: clientPath:null serverPath:null finished:false header:: 10,4  replyHeader:: 10,1430241685179,0  request:: '/hbase-secure/master,F  response:: #ffffffff000146d61737465723a31363030304affffff9232ffffffcb303137ffffffda50425546a28a1c6d6173746572312e6861646f6f702e74736c2e74656c75732e636f6d10ffffff807d18ffffff99ffffffe7ffffffffffffff81ffffff9d2b10018ffffff8a7d,s{1430236207048,1430236207048,1485257581917,1485257581917,0,0,0,241243744271518884,76,0,1430236207048} 
02-16-2017 16:56:08.403 ERROR ERP.hbase -  Reading reply sessionid:0x158d1da9640bb9a, packet:: clientPath:null serverPath:null finished:false header:: 11,3  replyHeader:: 11,1430241685180,0  request:: '/hbase-secure,F  response:: s{601295421598,601295421598,1450390322736,1450390322736,0,240,0,0,0,20,1430236207148} 
02-16-2017 16:56:08.406 ERROR ERP.hbase -  Reading reply sessionid:0x158d1da9640bb9a, packet:: clientPath:null serverPath:null finished:false header:: 12,4  replyHeader:: 12,1430241685180,0  request:: '/hbase-secure/master,F  response:: #ffffffff000146d61737465723a31363030304affffff9232ffffffcb303137ffffffda50425546a28a1c6d6173746572312e6861646f6f702e74736c2e74656c75732e636f6d10ffffff807d18ffffff99ffffffe7ffffffffffffff81ffffff9d2b10018ffffff8a7d,s{1430236207048,1430236207048,1485257581917,1485257581917,0,0,0,241243744271518884,76,0,1430236207048} 
02-16-2017 16:56:09.419 ERROR ERP.hbase -  Reading reply sessionid:0x158d1da9640bb9a, packet:: clientPath:null serverPath:null finished:false header:: 13,3  replyHeader:: 13,1430241685180,0  request:: '/hbase-secure,F  response:: s{601295421598,601295421598,1450390322736,1450390322736,0,240,0,0,0,20,1430236207148} 
02-16-2017 16:56:09.422 ERROR ERP.hbase -  Reading reply sessionid:0x158d1da9640bb9a, packet:: clientPath:null serverPath:null finished:false header:: 14,4  replyHeader:: 14,1430241685180,0  request:: '/hbase-secure/master,F  response:: #ffffffff000146d61737465723a31363030304affffff9232ffffffcb303137ffffffda50425546a28a1c6d6173746572312e6861646f6f702e74736c2e74656c75732e636f6d10ffffff807d18ffffff99ffffffe7ffffffffffffff81ffffff9d2b10018ffffff8a7d,s{1430236207048,1430236207048,1485257581917,1485257581917,0,0,0,241243744271518884,76,0,1430236207048} 
02-16-2017 16:56:11.426 ERROR ERP.hbase -  Reading reply sessionid:0x158d1da9640bb9a, packet:: clientPath:null serverPath:null finished:false header:: 15,3  replyHeader:: 15,1430241685184,0  request:: '/hbase-secure,F  response:: s{601295421598,601295421598,1450390322736,1450390322736,0,240,0,0,0,20,1430236207148} 
02-16-2017 16:56:11.428 ERROR ERP.hbase -  Reading reply sessionid:0x158d1da9640bb9a, packet:: clientPath:null serverPath:null finished:false header:: 16,4  replyHeader:: 16,1430241685184,0  request:: '/hbase-secure/master,F  response:: #ffffffff000146d61737465723a31363030304affffff9232ffffffcb303137ffffffda50425546a28a1c6d6173746572312e6861646f6f702e74736c2e74656c75732e636f6d10ffffff807d18ffffff99ffffffe7ffffffffffffff81ffffff9d2b10018ffffff8a7d,s{1430236207048,1430236207048,1485257581917,1485257581917,0,0,0,241243744271518884,76,0,1430236207048} 
02-16-2017 16:56:15.314 INFO  DispatchThread - Generating results preview took 1 ms
02-16-2017 16:56:15.445 ERROR ERP.hbase -  Reading reply sessionid:0x158d1da9640bb9a, packet:: clientPath:null serverPath:null finished:false header:: 17,3  replyHeader:: 17,1430241685187,0  request:: '/hbase-secure,F  response:: s{601295421598,601295421598,1450390322736,1450390322736,0,240,0,0,0,20,1430236207148} 
02-16-2017 16:56:15.447 ERROR ERP.hbase -  Reading reply sessionid:0x158d1da9640bb9a, packet:: clientPath:null serverPath:null finished:false header:: 18,4  replyHeader:: 18,1430241685187,0  request:: '/hbase-secure/master,F  response:: #ffffffff000146d61737465723a31363030304affffff9232ffffffcb303137ffffffda50425546a28a1c6d6173746572312e6861646f6f702e74736c2e74656c75732e636f6d10ffffff807d18ffffff99ffffffe7ffffffffffffff81ffffff9d2b10018ffffff8a7d,s{1430236207048,1430236207048,1485257581917,1485257581917,0,0,0,241243744271518884,76,0,1430236207048} 
02-16-2017 16:56:25.319 INFO  DispatchThread - Generating results preview took 1 ms
02-16-2017 16:56:25.451 ERROR ERP.hbase -  Reading reply sessionid:0x158d1da9640bb9a, packet:: clientPath:null serverPath:null finished:false header:: 19,3  replyHeader:: 19,1430241685197,0  request:: '/hbase-secure,F  response:: s{601295421598,601295421598,1450390322736,1450390322736,0,240,0,0,0,20,1430236207148} 
02-16-2017 16:56:25.452 ERROR ERP.hbase -  Got ping response for sessionid: 0x158d1da9640bb9a after 2ms
02-16-2017 16:56:25.454 ERROR ERP.hbase -  Reading reply sessionid:0x158d1da9640bb9a, packet:: clientPath:null serverPath:null finished:false header:: 20,4  replyHeader:: 20,1430241685197,0  request:: '/hbase-secure/master,F  response:: #ffffffff000146d61737465723a31363030304affffff9232ffffffcb303137ffffffda50425546a28a1c6d6173746572312e6861646f6f702e74736c2e74656c75732e636f6d10ffffff807d18ffffff99ffffffe7ffffffffffffff81ffffff9d2b10018ffffff8a7d,s{1430236207048,1430236207048,1485257581917,1485257581917,0,0,0,241243744271518884,76,0,1430236207048} 
02-16-2017 16:56:35.324 INFO  DispatchThread - Generating results preview took 1 ms
02-16-2017 16:56:35.518 ERROR ERP.hbase -  Reading reply sessionid:0x158d1da9640bb9a, packet:: clientPath:null serverPath:null finished:false header:: 21,3  replyHeader:: 21,1430241685205,0  request:: '/hbase-secure,F  response:: s{601295421598,601295421598,1450390322736,1450390322736,0,240,0,0,0,20,1430236207148} 
02-16-2017 16:56:35.518 ERROR ERP.hbase -  Got ping response for sessionid: 0x158d1da9640bb9a after 2ms
02-16-2017 16:56:35.520 ERROR ERP.hbase -  Reading reply sessionid:0x158d1da9640bb9a, packet:: clientPath:null serverPath:null finished:false header:: 22,4  replyHeader:: 22,1430241685205,0  request:: '/hbase-secure/master,F  response:: #ffffffff000146d61737465723a31363030304affffff9232ffffffcb303137ffffffda50425546a28a1c6d6173746572312e6861646f6f702e74736c2e74656c75732e636f6d10ffffff807d18ffffff99ffffffe7ffffffffffffff81ffffff9d2b10018ffffff8a7d,s{1430236207048,1430236207048,1485257581917,1485257581917,0,0,0,241243744271518884,76,0,1430236207048} 
02-16-2017 16:56:45.328 INFO  DispatchThread - Generating results preview took 1 ms
02-16-2017 16:56:45.555 ERROR ERP.hbase -  Reading reply sessionid:0x158d1da9640bb9a, packet:: clientPath:null serverPath:null finished:false header:: 23,3  replyHeader:: 23,1430241685213,0  request:: '/hbase-secure,F  response:: s{601295421598,601295421598,1450390322736,1450390322736,0,240,0,0,0,20,1430236207148} 
02-16-2017 16:56:45.555 ERROR ERP.hbase -  Got ping response for sessionid: 0x158d1da9640bb9a after 2ms
02-16-2017 16:56:45.557 ERROR ERP.hbase -  Reading reply sessionid:0x158d1da9640bb9a, packet:: clientPath:null serverPath:null finished:false header:: 24,4  replyHeader:: 24,1430241685213,0  request:: '/hbase-secure/master,F  response:: #ffffffff000146d61737465723a31363030304affffff9232ffffffcb303137ffffffda50425546a28a1c6d6173746572312e6861646f6f702e74736c2e74656c75732e636f6d10ffffff807d18ffffff99ffffffe7ffffffffffffff81ffffff9d2b10018ffffff8a7d,s{1430236207048,1430236207048,1485257581917,1485257581917,0,0,0,241243744271518884,76,0,1430236207048} 
02-16-2017 16:56:55.334 INFO  DispatchThread - Generating results preview took 1 ms
02-16-2017 16:56:55.579 ERROR ERP.hbase -  Reading reply sessionid:0x158d1da9640bb9a, packet:: clientPath:null serverPath:null finished:false header:: 25,3  replyHeader:: 25,1430241685225,0  request:: '/hbase-secure,F  response:: s{601295421598,601295421598,1450390322736,1450390322736,0,240,0,0,0,20,1430236207148} 
02-16-2017 16:56:55.579 ERROR ERP.hbase -  Got ping response for sessionid: 0x158d1da9640bb9a after 2ms
02-16-2017 16:56:55.581 ERROR ERP.hbase -  Reading reply sessionid:0x158d1da9640bb9a, packet:: clientPath:null serverPath:null finished:false header:: 26,4  replyHeader:: 26,1430241685225,0  request:: '/hbase-secure/master,F  response:: #ffffffff000146d61737465723a31363030304affffff9232ffffffcb303137ffffffda50425546a28a1c6d6173746572312e6861646f6f702e74736c2e74656c75732e636f6d10ffffff807d18ffffff99ffffffe7ffffffffffffff81ffffff9d2b10018ffffff8a7d,s{1430236207048,1430236207048,1485257581917,1485257581917,0,0,0,241243744271518884,76,0,1430236207048} 
02-16-2017 16:56:55.582 ERROR ERP.hbase -  Call exception, tries=10, retries=35, started=48397 ms ago, cancelled=false, msg=

[jruaux_splunk]

Hi,

Could you share your inputs.conf? Also, does your secure cluster look like this: http://www.cloudera.com/documentation/enterprise/5-5-x/topics/cdh_sg_hbase_authentication.html

Thanks,
Julien

[eangeles]

Hi Julien - are you referring to the inputs.conf file in $SPLUNKHOME/etc/system/default ? The inputs.conf file in /etc/system/local has only one entry:

[default]
host = splunk.hadoop.companyABC.com

[jruaux_splunk]

Sorry I meant indexes.conf (that should contain your HBase connection settings)

[eangeles]

This is the file in $SPLUNKHOME/etc/apps/hbase-erp/local/indexes.conf:

[provider:hbase]
vix.env.HADOOP_HOME = /usr/hdp/current/hadoop-client
vix.env.HBASE_HOME = /usr/hdp/current/hbase-client
vix.env.JAVA_HOME = /usr/jdk64/jdk1.7.0_67
vix.hbase.zookeeper.quorum = master1.hadoop.companyABC.com,slave1.hadoop.companyABC.com,master2.hadoop.companyABC.com
vix.output.buckets.max.network.bandwidth = 0
vix.zookeeper.znode.parent = /hbase-secure
vix.loglevel = ALL
vix.splunk.search.debug = 0
vix.dfs.namenode.kerberos.principal = nn/_HOST@HADOOP.COMPANYABC.COM
vix.hadoop.security.authentication = kerberos
vix.hadoop.security.authorization = 1
vix.kerberos.keytab = /etc/security/keytabs/poweruser.keytab
vix.kerberos.principal = sauron@HADOOP.COMPANYABC.COM
vix.yarn.nodemanager.principal = nm/_HOST@HADOOP.COMPANYABC.COM
vix.yarn.resourcemanager.principal = rm/_HOST@HADOOP.COMPANYABC.COM
vix.javaprops.java.security.krb5.conf = /etc/krb5.conf

[eangeles]

Was there anything else that I am missing from the configuration standpoint?

[jruaux_splunk]

In your provider-family stanza, could you try adding this parameter (as specified in https://www.cloudera.com/documentation/enterprise/5-5-x/topics/cdh_sg_hbase_authentication.html)

[provider-family:hbase_family]
vix.mode = stream
vix.command =  java
vix.command.arg.1 = -Xmx512m
vix.command.arg.2 = -classpath
vix.command.arg.3 = $SPLUNK_HOME/bin/jars/SplunkMR-h1.jar:$SPLUNK_HOME/etc/apps/${project.artifactId}/bin/${project.artifactId}-${project.version}.jar:$SPLUNK_HOME/etc/apps/${project.artifactId}/bin/lib/*
vix.command.arg.4 = -Djava.security.auth.login.config=/etc/hbase/conf/zk-jaas.conf
vix.command.arg.5 = com.splunk.erp.hbase.HBaseERP

where /etc/hbase/conf/zk-jaas.conf contains the following:

   Client {
      com.sun.security.auth.module.Krb5LoginModule required
      useKeyTab=true
      useTicketCache=false
      keyTab="/etc/hbase/conf/hbase.keytab"
      principal="hbase/fully.qualified.domain.name@<YOUR-REALM>";
   };