Thanks Clint! Will work on tweaking the inputs per the note above ... View more

There is a setup video on this page to setup Hunk with HDP 1.1.3 (all the way at the bottom): http://hortonworks.com/hadoop-tutorial/configure-hortonworks-sandbox-version-2-0-with-hunk-splunk-analytics-for-hadoop/ The configurations to setup Hunk with HDP 2.1 is slightly different (due to Yarn). Here is my configurations based on HDP 2.1 Sandbox: [provider:hunk62hdprovider] vix.command.arg.3 = $SPLUNK_HOME/bin/jars/SplunkMR-s6.0-hy2.0.jar vix.description = Hunk 62 Hortonworks 2.1 Provider vix.env.HADOOP_HOME = /usr vix.env.JAVA_HOME = /usr vix.family = hadoop vix.fs.default.name = hdfs://sandbox:8020 vix.mapreduce.framework.name = yarn vix.splunk.home.hdfs = /user/root/hunk62mr vix.yarn.resourcemanager.address = sandbox:8050 vix.yarn.resourcemanager.scheduler.address = sandbox:8030 ... View more

設定ファイル streamfwd.xml 内のPortを0に指定することで可能です。 有効にしておく場合、管理画面への不要なアクセスを防ぐためにもiptable等でアクセス制限をしたほうがいいですね。 Disable Stream Forwarder admin interface By default, the Stream Forwarder admin interface is enabled, listening on TCP port 8889. To disable the interface: 1. Go to $SPLUNK_HOME/etc/apps/Splunk_TA_stream/local. 2. Open the streamfwd.xml configuration file and change 8889 to 0. 参考：ドキュメントのここ ... View more

I took this JSON file = Hunkdata.json Before Splunk Indexing: 671 MB After Splunk Indexing (raw data + Index data): 463 MB = About 70% of original file After Archiving it into HDFS (raw data + few metadata files): 157 MB = About 33% of Splunk indexer ... View more

Nice, thanks for the update. We'll try it out. ... View more

I was running hunk from a linux host that didnt have hdfs access to the remote cluster. Rather than try and sort that access out I installed hunk on the cluster and it runs ok. ... View more

The streamfwd binary included in the App for Stream TA (Splunk_TA_stream) is a modular input. Like any other modular input, it runs as a subprocess of splunkd and writes all events to stdout. It is completely up to splunkd (your heavy or universal forwarder) where and how the data is sent to indexers. Stream plays no part in it. ... View more

Were you able to edit the streamfwd.xml config file in the local directory for your Splunk Stream instance? Check out this link, it goes over setting up Splunk Stream to look at a specific PCAP file : https://answers.splunk.com/answers/220040/running-stream-against-old-pcaps.html I hope that helps! ... View more

It worked once I added the indexer name and port 8000 just like goancea did. ... View more

It's very likely that newer versions of CentOS, RHEL, Ubuntu and Debian should all work fine with App for Stream. The current list includes platforms that we have tested it on. We recognize that it is dated, and plan to add newer versions to our test matrix soon. ... View more

Also don't forget to run sudo ./setuid from the Splunk_TA_Stream directory. ... View more

Thanks for looking in, you are right, I was using the wrong port. I tested using an incorrect IP and saw it error, but I didn't think to test the port. I am now in, the tool looks great. ... View more

I installed the trial version for splunk&hunk without the license. Now I'm trying to follow the tutorial in Hunk-6.0-Hunk_User_Manual. I'm trying to perform step 5: configuring an HDFS provider. I can't see the option virtual indexes under settings -> data What can be the problem? ... View more

Yes. Thanks for pointing to the removal of the field. ... View more

thanks very much. this info will be very useful for me ... View more

Google developed Snappy algorithm in c : https://code.google.com/p/snappy/ which is ported in java by various third party. For more information, please see http://blog.cloudera.com/blog/2011/09/snappy-and-hadoop/. Hadoop shipped with https://code.google.com/p/hadoop-snappy/ which is missing in your hadoop client classpath. In my apache hadoop 2.0.5-alpha, the SnappyCodec class is in hadoop-common-2.0.5-alpha.jar [hyan@ronnie hunk-staging]$ jar -J-Xmx512m -tf /mnt/big/hyan/hadoop/hadoop-2.0.5-alpha/share/hadoop/common/hadoop-common-2.0.5-alpha.jar |grep Snappy org/apache/hadoop/io/compress/snappy/SnappyDecompressor.class org/apache/hadoop/io/compress/snappy/SnappyCompressor.class org/apache/hadoop/io/compress/SnappyCodec.class Here is my hadoop classpath: [hyan@ronnie hunk-staging]$ /mnt/big/hyan/hadoop/hadoop-2.0.5-alpha/bin/hadoop classpath | sed 's/:/\n/g' /mnt/big/hyan/hadoop/hadoop-2.0.5-alpha/etc/hadoop /mnt/big/hyan/hadoop/hadoop-2.0.5-alpha/share/hadoop/common/lib/* /mnt/big/hyan/hadoop/hadoop-2.0.5-alpha/share/hadoop/common/* /contrib/capacity-scheduler/*.jar /mnt/big/hyan/hadoop/hadoop-2.0.5-alpha/share/hadoop/hdfs /mnt/big/hyan/hadoop/hadoop-2.0.5-alpha/share/hadoop/hdfs/lib/* /mnt/big/hyan/hadoop/hadoop-2.0.5-alpha/share/hadoop/hdfs/* /mnt/big/hyan/hadoop/hadoop-2.0.5-alpha/share/hadoop/yarn/lib/* /mnt/big/hyan/hadoop/hadoop-2.0.5-alpha/share/hadoop/yarn/* /mnt/big/hyan/hadoop/hadoop-2.0.5-alpha/share/hadoop/mapreduce/lib/* /mnt/big/hyan/hadoop/hadoop-2.0.5-alpha/share/hadoop/mapreduce/* The snappy java library uses the native c library, therefore, those native libraries need to be on native library path (set by 'java.library.path' java system property). This system property is set by one of the hadoop config files before calling bin/hadoop script and could be reset by user. The default location seems to be different between various hadoop version/distro. For hadoop 1.0.3, as Ledion shown, it is under $HADOOP_HOME/lib/native/ /. For hadoop 2.0.5, it is under $HADOOP_HOME/lib/native/. You can reset the path in hunk by adding the following line to indexes.conf: vix.env.HADOOP_CLIENT_OPTS = -Djava.library.path=/mnt/big/hyan/local/lib/ Note, if you installed your hadoop client from a hadoop tar file on your search head, the native hadoop library ($HADOOP_HOME/lib/native/libhadoop.so) most likely does not work with snappy (because it is an optional component). You may need to rebuild the hadoop native library together with snappy component. Here is my native lib directory: [hyan@ronnie hunk-staging]$ ls -l /mnt/big/hyan/local/lib/ total 66028 -rw-r--r-- 1 hyan games 124788 Oct 3 16:42 libcontainer.a -rw-r--r-- 1 hyan games 753980 Oct 3 16:42 libhadoop.a -rw-r--r-- 1 hyan games 1482396 Oct 3 16:42 libhadooppipes.a -rwxr-xr-x 1 hyan games 412640 Oct 3 16:42 libhadoop.so -rwxr-xr-x 1 hyan games 412640 Oct 3 16:42 libhadoop.so.1.0.0 -rw-r--r-- 1 hyan games 579600 Oct 3 16:42 libhadooputils.a -rw-r--r-- 1 hyan games 268948 Oct 3 16:42 libhdfs.a -rwxr-xr-x 1 hyan games 177913 Oct 3 16:42 libhdfs.so -rwxr-xr-x 1 hyan games 177913 Oct 3 16:42 libhdfs.so.0.0.0 -rw-r--r-- 1 hyan games 44066 Oct 3 16:42 libnative_mini_dfs.a -rw-r--r-- 1 hyan games 17230 Oct 3 16:42 libposix_util.a -rw-r--r-- 1 hyan games 17648118 Oct 2 15:34 libprotobuf.a -rwxr-xr-x 1 hyan games 1003 Oct 2 15:34 libprotobuf.la -rw-r--r-- 1 hyan games 1947834 Oct 2 15:34 libprotobuf-lite.a -rwxr-xr-x 1 hyan games 1038 Oct 2 15:34 libprotobuf-lite.la lrwxrwxrwx 1 hyan games 25 Oct 2 15:34 libprotobuf-lite.so -> libprotobuf-lite.so.7.0.0 lrwxrwxrwx 1 hyan games 25 Oct 2 15:34 libprotobuf-lite.so.7 -> libprotobuf-lite.so.7.0.0 -rwxr-xr-x 1 hyan games 893019 Oct 2 15:34 libprotobuf-lite.so.7.0.0 lrwxrwxrwx 1 hyan games 20 Oct 2 15:34 libprotobuf.so -> libprotobuf.so.7.0.0 lrwxrwxrwx 1 hyan games 20 Oct 2 15:34 libprotobuf.so.7 -> libprotobuf.so.7.0.0 -rwxr-xr-x 1 hyan games 7324725 Oct 2 15:34 libprotobuf.so.7.0.0 -rw-r--r-- 1 hyan games 25882136 Oct 2 15:34 libprotoc.a -rwxr-xr-x 1 hyan games 1028 Oct 2 15:34 libprotoc.la lrwxrwxrwx 1 hyan games 18 Oct 2 15:34 libprotoc.so -> libprotoc.so.7.0.0 lrwxrwxrwx 1 hyan games 18 Oct 2 15:34 libprotoc.so.7 -> libprotoc.so.7.0.0 -rwxr-xr-x 1 hyan games 9071244 Oct 2 15:34 libprotoc.so.7.0.0 -rw-r--r-- 1 hyan games 207892 Oct 3 16:42 libsnappy.a -rwxr-xr-x 1 hyan games 962 Oct 3 16:42 libsnappy.la lrwxrwxrwx 1 hyan games 18 Oct 2 15:42 libsnappy.so -> libsnappy.so.1.1.4 lrwxrwxrwx 1 hyan games 18 Oct 2 15:42 libsnappy.so.1 -> libsnappy.so.1.1.4 -rwxr-xr-x 1 hyan games 128527 Oct 3 16:42 libsnappy.so.1.1.4 drwxr-xr-x 2 hyan games 4096 Oct 2 15:34 pkgconfig you can verify if snappy already works with your hadoop client by: $HADOOP_HOME/bin/hadoop fs -text s3n://AKIAI5CIAYX6LNOC6XMQ:4PqCDiaAdPoOR2puiq9cjHv7PtxoMF3cIW0GRsqO@wp-dw-source/omni/site=washpostcom/dt=20090616/a088a0a4-3a18-4dae-a551-ecf25a855367_000389.snappy ... View more

Hi every one. Finally this problem seems to be resolved. Our hadoop machines are not listed on our DNS, we are using /etc/hosts to asign a name to them. It seems that the aplication is issuing a DNS request for the machine name and not getting the name from /etc/hosts. All the Hadopo commands and processes use /etc/hosts normally. We have configured the Hadoop URL using headnode's IP and it register the cluster. ... View more

I've just tried - with success - creating the following saved search : my_search | bin _time span=5m | sistats count by _time interesting_field which is then used as follows : my_search | bin _time span=5m | sistats count by _time interesting_field | timechart span=1h count by interesting_field The acceleration on the initial saved search did give me a big boost in the final search - I'm timecharting about 500 million events in less than 30 seconds now. You can choose the span in the timechart as you like, but sensible results are only obtained by using a multiple of the initial time bin span. ... View more

Yes! https://github.com/splunk/eventgen . Help yourself. It's very well documented, and you can email me personally (email address should be easily guessable) or find me on IRC (Coccyx) to get help. ... View more

Agree with Clint and adding a high level summary. Thanks to Adrian for this...just wanted to make sure it gets shared. SCOM What SCOM is good at: - Performance Monitoring - Single system Alert Management (rules based identification of problems on a single host) What SCOM is not very good at: - Root Cause Analysis via machine data ( based on a SQL data store, event viewer type functionality) - Historical Trending of Log Information (based on a SQL data store) - Multi system or complex alerts (rules based identification of problems based on logs across multiple hosts) SPLUNK: What SPLUNK is good at: - Root Cause Analysis via machine data - Historical Trending of Log Information - Multi system or complex alerts (rules based identification of problems based on logs across multiple hosts) What SPLUNK is not very good at: - Performance Monitoring - Rules based identification of problems on a single host using non-gathered data ... View more

Okay. I understand now. I won't ask the question anymore. ... View more