topic Re: Why am I getting this error in the search.log when extracting Hive data in Splunk? in Splunk Search

Why am I getting this error in the search.log when extracting Hive data in Splunk?

maximus_reborn — Tue, 26 Apr 2016 11:35:28 GMT

I am getting the below error in the search.log when I am extracting hive data in Splunk.
I am using thrift metastore of hive while extracting data in Splunk.

Caused by: java.lang.VerifyError: class org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$CompleteRequestProto overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet

Re: Why am I getting this error in the search.log when extracting Hive data in Splunk?

kschon_splunk — Tue, 29 Sep 2020 09:29:59 GMT

That's looks like a version mis-match error. Can you confirm which version Hive version your metastore is? And can you paste here the vix.env.HUNK_THIRDPARTY_JARS property from the provider you are using to search with?

Re: Why am I getting this error in the search.log when extracting Hive data in Splunk?

maximus_reborn — Tue, 29 Sep 2020 09:30:06 GMT

Are you asking about the below information. Otherwise Please let me know how to check the metastore version.
$ schematool -dbType mysql -info
Metastore connection URL: jdbc:mysql://127.0.0.1:3306/hive?createDatabaseIfNotExist=true
Metastore Connection Driver : com.mysql.jdbc.Driver
Metastore connection User: root
Hive distribution version: 1.2.0
Metastore schema version: 1.2.0

I don't have any property of vix.env.HUNK_THIRDPARTY_JARS. 😞 What value should it has?

Re: Why am I getting this error in the search.log when extracting Hive data in Splunk?

kschon_splunk — Tue, 29 Sep 2020 09:34:00 GMT

It looks like your Hive version is 1.2.0. If vix.env.HUNK_THIRDPARTY_JARS is not specified in your provider, then you are using the default value, which probably assumes Hive version 0.12. Which Hadoop distribution are you using, and which version? And which version of Hunk are you using?

Re: Why am I getting this error in the search.log when extracting Hive data in Splunk?

burwell — Tue, 29 Sep 2020 09:34:05 GMT

Keith, it seems this is a bug with Splunk 6.4. The etc/system/default/indexes.conf has the Hive 0.12 binaries.

vix.env.HUNK_THIRDPARTY_JARS = $SPLUNK_HOME/bin/jars/thirdparty/common/avro-1.7.7.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/avro-mapred-1.7.7.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/commons-compress-1.10.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/commons-io-2.4.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/libfb303-0.9.2.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/parquet-hive-bundle-1.5.0.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/snappy-java-1.1.1.7.jar,$SPLUNK_HOME/bin/jars/thirdparty/hive/hive-exec-0.12.0.jar,$SPLUNK_HOME/bin/jars/thirdparty/hive/hive-metastore-0.12.0.jar,$SPLUNK_HOME/bin/jars/thirdparty/hive/hive-serde-0.12.0.jar

Re: Why am I getting this error in the search.log when extracting Hive data in Splunk?

kschon_splunk — Tue, 29 Sep 2020 09:34:08 GMT

Hi Becky. That's correct. The default value still assumes you are using a Hadoop 1.x distribution, most of which came with Hive 0.x. If you create a new provider and change the drop-down to Yarn, it will change the value of that property to this:

vix.env.HUNK_THIRDPARTY_JARS = $SPLUNK_HOME/bin/jars/thirdparty/common/avro-1.7.7.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/avro-mapred-1.7.7.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/commons-compress-1.10.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/commons-io-2.4.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/libfb303-0.9.2.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/parquet-hive-bundle-1.5.0.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/snappy-java-1.1.1.7.jar,$SPLUNK_HOME/bin/jars/thirdparty/hive_1_2/hive-exec-1.2.1.jar,$SPLUNK_HOME/bin/jars/thirdparty/hive_1_2/hive-metastore-1.2.1.jar,$SPLUNK_HOME/bin/jars/thirdparty/hive_1_2/hive-serde-1.2.1.jar

This value refers to Hive 1.2.1. The reason I asked about the Hadoop version is that, unfortunately, Hive 1.x changed some method signatures, so the Splunk jar you use must be compiled against both the Hadoop version (1.x or 2.x) and the Hive version (0.x or 1.x).

Assuming that you are using Hadoop 2.x, you should be able to just add the above line to you provider stanza, and hopefully that will fix your problem. Or you could change it to point to your own Hive jars (it looks like you are using 1.2.0, and Hunk 6.4 ships with 1.2.1), but I don't think it will be necessary.

Re: Why am I getting this error in the search.log when extracting Hive data in Splunk?

maximus_reborn — Thu, 28 Apr 2016 16:46:11 GMT

Hi Kschon, I am using Hadoop 2.6 and Hive 1.2.1. I am not using any hadoop distribution. I have installed its binary from the apache mirror servers. I have done this with Hive also. Have i answered your question?

Re: Why am I getting this error in the search.log when extracting Hive data in Splunk?

maximus_reborn — Thu, 28 Apr 2016 16:58:22 GMT

Thanks kschon. That solved the problem.

Re: Why am I getting this error in the search.log when extracting Hive data in Splunk?

kschon_splunk — Thu, 28 Apr 2016 18:25:18 GMT

Great! If there are no other issues, can you "accept" the answer?