Are you able to run an example MR job with the principal from hadoop cli on Splunk Search Head? ... View more

The default port number for resource manager is 8032, please make sure your cluster setting is changed to 8050. yarn.resourcemanager.address = :8050 Also, please make sure the hdfs.headless.keytab file is saved on /etc/security/keytabs/ directory of your Splunk Search Head and the principal name is correct. vix.kerberos.principal = hdfs@ ... View more

Yes. vix.input.1.splitter.hive.rowformat.fields.terminated=, ... View more

A better way is to use index time extraction and force your time field to be a required field. So in the vix stanza of indexes.conf [vix] vix.input.1.required.fields = yourTimeField and then use props.conf to configure index time timestamp extraction [sourcetype] TIME_PREFIX="yourTimeField": TIME_FORMAT = %s The makes time-based partition pruning work because: a) we assume that the files within a directory contain time from the range extracted from the path b) the value of _time is known to be coming from the events (ie not modified), which due to (a) have to be within the time range of the path ... View more

Hunk converts all hive data to json format. The following line disables index-time timestamping: DATETIME_CONFIG = NONE To convert your epoch format time field, do this: EVAL-_time = strptime(yourTimeField, "%s") Here is various strptime formats supported by Splunk http://docs.splunk.com/Documentation/Splunk/latest/Data/Configuretimestamprecognition#Enhanced_strptime.28.29_support ... View more

I suspect hive-0.14 jar is still picked up by Hunk. You can turn on debug mode by setting "vix.splunk.search.debug=1" in provider config, run a search and check search.log. Looking for these two lines: 05-10-2015 21:49:53.499 DEBUG ERP. - HADOOP_CLASSPATH=... 05-10-2015 21:49:55.109 INFO ERP. - SplunkMR - Setting custom jars=... and make sure your hive-0.12 or 0.13 jars are on the lists and before hive-0.14 jars if there is any on the lists. If they are, then you should open a support case and upload a diag. ... View more

I assume you replaced the hive-0.12 jars Hunk shipped with your hive-0.14 ones. This problem is due to hive-0.14 introduced a non-backward compatible API. I will file a bug and fix it later. You have two workarounds: 1) set property "vix.splunk.search.splitter.hive.ppd=0" in your provider stanza. However, this will turn off ORC Predicate Pushdown feature. 2) replace hive-0.14 jars specified in Hunk vix.env.HUNK_THIRDPARTY_JARS with lower version hive to avoid turning off ORC predicate pushdown. Note, if your table column defined CHAR type introduced at 0.13, you can only downgrade to 0.13. ... View more

vix.splunk.heartbeat.threshold - number of heartbeat delayed by search head before MR job commits suicide vix.splunk.heartbeat.interval - how often does search head heartbeat heartbeat is achieved by renaming a heartbeat file from search head. The default 1 second interval assumes your file system rename operation can be done in a second. If that’s not the case, e.g. s3n system, or the network connection between search head to Hadoop NameNode is slow, or simply because your Hadoop cluster is too busy to react, you should increase the interval. On the other hand, you could decrease heartbeat threshold to make Hunk kill run away MR job more promptly. ... View more

For any hive formats other than text, you need to specify fileformat property. So in your case: vix.input.1.splitter.hive.fileformat = orc ... View more

For any hive formats other than text, you need to specify fileformat property. So in your case: vix.input.1.splitter.hive.fileformat = orc ... View more

It might because your export job start time (the "Export from" item in export create/edit page) is set too early and the first one or few runs were not getting any events. You can manually run the job a few times (by clicking the 'run' link in the job list page) and see if you get any event exported. You can also check the HadoopConnect.log under $SPLUNK_HOME/var/log/splunk/. Each time export is run, you will get two lines like this: 2014-04-14 13:20:14,868 INFO run_export.py [main] [878] - search:search (_indextime=1397494147 OR _indextime=1397494148 ... OR _indextime=1397495946) host=foo| eval _dstpath=strftime(_time, "%Y%m%d") + "/" + strftime(_time, "%H") + "/" + host | fields _dstpath host source host,source,sourcetype | dump basefilename=foobar ... exportname="Host" 2014-04-14 13:21:51,304 INFO run_export.py [renameTmpFiles] [354] - Renaming 2 temporary files in HDFS... You can run the first part of the search query in search bar and verify if there is any events fall into that export time frame: (_indextime=1397494147 OR _indextime=1397494148 ... OR _indextime=1397495946) host=foo| eval _dstpath=strftime(_time, "%Y%m%d") + "/" + strftime(_time, "%H") + "/" + host | fields _dstpath host source host,source,sourcetype The second log statement tells you how many event files are exported to HDFS. ... View more

Google developed Snappy algorithm in c : https://code.google.com/p/snappy/ which is ported in java by various third party. For more information, please see http://blog.cloudera.com/blog/2011/09/snappy-and-hadoop/. Hadoop shipped with https://code.google.com/p/hadoop-snappy/ which is missing in your hadoop client classpath. In my apache hadoop 2.0.5-alpha, the SnappyCodec class is in hadoop-common-2.0.5-alpha.jar [hyan@ronnie hunk-staging]$ jar -J-Xmx512m -tf /mnt/big/hyan/hadoop/hadoop-2.0.5-alpha/share/hadoop/common/hadoop-common-2.0.5-alpha.jar |grep Snappy org/apache/hadoop/io/compress/snappy/SnappyDecompressor.class org/apache/hadoop/io/compress/snappy/SnappyCompressor.class org/apache/hadoop/io/compress/SnappyCodec.class Here is my hadoop classpath: [hyan@ronnie hunk-staging]$ /mnt/big/hyan/hadoop/hadoop-2.0.5-alpha/bin/hadoop classpath | sed 's/:/\n/g' /mnt/big/hyan/hadoop/hadoop-2.0.5-alpha/etc/hadoop /mnt/big/hyan/hadoop/hadoop-2.0.5-alpha/share/hadoop/common/lib/* /mnt/big/hyan/hadoop/hadoop-2.0.5-alpha/share/hadoop/common/* /contrib/capacity-scheduler/*.jar /mnt/big/hyan/hadoop/hadoop-2.0.5-alpha/share/hadoop/hdfs /mnt/big/hyan/hadoop/hadoop-2.0.5-alpha/share/hadoop/hdfs/lib/* /mnt/big/hyan/hadoop/hadoop-2.0.5-alpha/share/hadoop/hdfs/* /mnt/big/hyan/hadoop/hadoop-2.0.5-alpha/share/hadoop/yarn/lib/* /mnt/big/hyan/hadoop/hadoop-2.0.5-alpha/share/hadoop/yarn/* /mnt/big/hyan/hadoop/hadoop-2.0.5-alpha/share/hadoop/mapreduce/lib/* /mnt/big/hyan/hadoop/hadoop-2.0.5-alpha/share/hadoop/mapreduce/* The snappy java library uses the native c library, therefore, those native libraries need to be on native library path (set by 'java.library.path' java system property). This system property is set by one of the hadoop config files before calling bin/hadoop script and could be reset by user. The default location seems to be different between various hadoop version/distro. For hadoop 1.0.3, as Ledion shown, it is under $HADOOP_HOME/lib/native/ /. For hadoop 2.0.5, it is under $HADOOP_HOME/lib/native/. You can reset the path in hunk by adding the following line to indexes.conf: vix.env.HADOOP_CLIENT_OPTS = -Djava.library.path=/mnt/big/hyan/local/lib/ Note, if you installed your hadoop client from a hadoop tar file on your search head, the native hadoop library ($HADOOP_HOME/lib/native/libhadoop.so) most likely does not work with snappy (because it is an optional component). You may need to rebuild the hadoop native library together with snappy component. Here is my native lib directory: [hyan@ronnie hunk-staging]$ ls -l /mnt/big/hyan/local/lib/ total 66028 -rw-r--r-- 1 hyan games 124788 Oct 3 16:42 libcontainer.a -rw-r--r-- 1 hyan games 753980 Oct 3 16:42 libhadoop.a -rw-r--r-- 1 hyan games 1482396 Oct 3 16:42 libhadooppipes.a -rwxr-xr-x 1 hyan games 412640 Oct 3 16:42 libhadoop.so -rwxr-xr-x 1 hyan games 412640 Oct 3 16:42 libhadoop.so.1.0.0 -rw-r--r-- 1 hyan games 579600 Oct 3 16:42 libhadooputils.a -rw-r--r-- 1 hyan games 268948 Oct 3 16:42 libhdfs.a -rwxr-xr-x 1 hyan games 177913 Oct 3 16:42 libhdfs.so -rwxr-xr-x 1 hyan games 177913 Oct 3 16:42 libhdfs.so.0.0.0 -rw-r--r-- 1 hyan games 44066 Oct 3 16:42 libnative_mini_dfs.a -rw-r--r-- 1 hyan games 17230 Oct 3 16:42 libposix_util.a -rw-r--r-- 1 hyan games 17648118 Oct 2 15:34 libprotobuf.a -rwxr-xr-x 1 hyan games 1003 Oct 2 15:34 libprotobuf.la -rw-r--r-- 1 hyan games 1947834 Oct 2 15:34 libprotobuf-lite.a -rwxr-xr-x 1 hyan games 1038 Oct 2 15:34 libprotobuf-lite.la lrwxrwxrwx 1 hyan games 25 Oct 2 15:34 libprotobuf-lite.so -> libprotobuf-lite.so.7.0.0 lrwxrwxrwx 1 hyan games 25 Oct 2 15:34 libprotobuf-lite.so.7 -> libprotobuf-lite.so.7.0.0 -rwxr-xr-x 1 hyan games 893019 Oct 2 15:34 libprotobuf-lite.so.7.0.0 lrwxrwxrwx 1 hyan games 20 Oct 2 15:34 libprotobuf.so -> libprotobuf.so.7.0.0 lrwxrwxrwx 1 hyan games 20 Oct 2 15:34 libprotobuf.so.7 -> libprotobuf.so.7.0.0 -rwxr-xr-x 1 hyan games 7324725 Oct 2 15:34 libprotobuf.so.7.0.0 -rw-r--r-- 1 hyan games 25882136 Oct 2 15:34 libprotoc.a -rwxr-xr-x 1 hyan games 1028 Oct 2 15:34 libprotoc.la lrwxrwxrwx 1 hyan games 18 Oct 2 15:34 libprotoc.so -> libprotoc.so.7.0.0 lrwxrwxrwx 1 hyan games 18 Oct 2 15:34 libprotoc.so.7 -> libprotoc.so.7.0.0 -rwxr-xr-x 1 hyan games 9071244 Oct 2 15:34 libprotoc.so.7.0.0 -rw-r--r-- 1 hyan games 207892 Oct 3 16:42 libsnappy.a -rwxr-xr-x 1 hyan games 962 Oct 3 16:42 libsnappy.la lrwxrwxrwx 1 hyan games 18 Oct 2 15:42 libsnappy.so -> libsnappy.so.1.1.4 lrwxrwxrwx 1 hyan games 18 Oct 2 15:42 libsnappy.so.1 -> libsnappy.so.1.1.4 -rwxr-xr-x 1 hyan games 128527 Oct 3 16:42 libsnappy.so.1.1.4 drwxr-xr-x 2 hyan games 4096 Oct 2 15:34 pkgconfig you can verify if snappy already works with your hadoop client by: $HADOOP_HOME/bin/hadoop fs -text s3n://AKIAI5CIAYX6LNOC6XMQ:4PqCDiaAdPoOR2puiq9cjHv7PtxoMF3cIW0GRsqO@wp-dw-source/omni/site=washpostcom/dt=20090616/a088a0a4-3a18-4dae-a551-ecf25a855367_000389.snappy ... View more