About Ledion_Bitincka

Ledion_Bitincka · ‎07-18-2014

Are you able to use the Hadoop CLI to access the s3 filesystem (hadoop fs -ls s3://... )?

Ledion_Bitincka · ‎07-07-2014

There are no predefined hooks in the interface. However, you should be able to detect when a search process has been paused / finalized / canceled by periodically looking for sentinel files (with the same names) in the search's dispatch dir $SPLUNK_HOME/var/run/splunk/dispatch/<sid> The search process when: paused: will stop reading from the stdin pipe (and the ERP's stdout writes will block until unpaused) finalized / canceled: will close the stdin pipe (and ERP's stdout writes will fail with broken pipe)

Ledion_Bitincka · ‎07-07-2014

You should be able to use a few of the methods for extract fields at search time: Interactive field extractor Config files (props/transforms.conf)

Ledion_Bitincka · ‎06-27-2014

You can either email them to support or maybe post them on pastebin and provide a link

Ledion_Bitincka · ‎06-27-2014

Can you describe what you want the timechart to look like? I'd guess there will be two series: one showing the distinct count, hourly and another showing the max at the daily granularity?

Ledion_Bitincka · ‎06-25-2014

Can you please provide the (scrubbed) contents of search.log as well as indexes.conf? Also can you test to see if the following command throws the same error: hadoop fs -text hdfs://host:port/path/to/file.snappy Also, what version of Hadoop and Snappy libraries are you using?

Ledion_Bitincka · ‎06-23-2014

The following settings have no effect on Hunk as we use gzip (not configurable) to compress the mapper output results. vix.mapreduce.output.fileoutputformat.compress.codec = org.apache.hadoop.io.compress.SnappyCodec vix.mapreduce.output. fileoutputformat.compress.type = BLOC vix.mapreduce.output.fileoutputformat.compress = true So using a date range and verbose mode, it takes about 10 minutes to process 94 files X 300MB per file. Any particular reason that you're using verbose mode? This search mode is used primarily for exploratory needs and it is extremely expensive for reporting searches and introduces quite a bit of over head. If these need to be compressed on HDFS I assume LZO or Snappy? Yes, there's usually performance benefits from using compressed raw data - I'd recommend Snappy as it generally has better read throughput performance. What types of searches are you trying to run on the data?

Ledion_Bitincka · ‎06-10-2014

No, Hunk currently only runs on Linux and supports Hadoop running on Linux

Ledion_Bitincka · ‎05-27-2014

In 6.1 Hunk and Splunk are two separate packages, thus Hunk functionality cannot be enabled by simply adding a license to a Splunk Enterprise package.

Ledion_Bitincka · ‎05-27-2014

Hunk does not create any persistent artifacts or index after running a search, ie it does "index" the data after first search. However, for searches that you intend to run frequently in Hunk 6.1 you can enable report acceleration to cache the intermediate results so they are reused next time Here's another related question - not sure if it's the same question from two co-workers 🙂

Ledion_Bitincka · ‎05-27-2014

You can either extract the time range from the parent dir: Time Capturing Regex: "/user/netflow/(d+)-(d+)-(d+)/" Time Format: "yyyyMMdd" Time Adjustment: 0 Time Range: 1 day or your can extract the more granular timestamp at the file level: Time Capturing Regex: "asciiflow(\d+)-(\d+)-(\d+)-(\d+)-(\d+)-\d+.csv$" Time Format: "yyyyMMddHHmm" Time Adjustment: 0 Time Range: 15 minutes

Ledion_Bitincka · ‎05-27-2014

Just to clarify, Hunk does not create an index based on the data that it searches once. Yes, we do recommend that you partition your data based on time and any other fields that you search frequently.

Ledion_Bitincka · ‎05-19-2014

Ok, does that mean that the RM settings for the provider are incorrect then? What does your indexes.conf look like?

Ledion_Bitincka · ‎05-19-2014

Is RM running on port 8050? The search without the head command causes Hunk to spawn a MapReduce/Yarn job thus the need for communicating with RM.

Ledion_Bitincka · ‎05-19-2014

Is the resource manager running on the localhost? The following log messages indicate that we can't connect to the RM - either it is not running, not responding or we've got the wrong host 05-16-2014 09:39:59.630 INFO ERP.TMob_Core_Metrics - Client$Connection - Retrying connect to server: localhost:8050. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 05-16-2014 09:40:00.631 INFO ERP.TMob_Core_Metrics - Client$Connection - Retrying connect to server: localhost:8050. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) yet I see that HDFS is running elsewhere 05-16-2014 09:39:58.484 INFO ERP.TMob_Core_Metrics - SplunkMR$SplunkBaseMapper - using class=com.splunk.mr.input.SplunkLineRecordReader to process split=hdfs://RHEL6:9000/user/hadoop2/trace_log:0+134217728

Ledion_Bitincka · ‎05-19-2014

Can you please send us a link to the contents of search.log? You can get to it by clicking on "Job" dropdown (above timeline to the right) >> "Inspect Job" >> "search.log" (bottom of page)

Ledion_Bitincka · ‎05-12-2014

Hmmm, since events are not guaranteed to be processed in any particular order and you absolutely need order to match the most recent start with the correct stop event (to compute the duration) there's no way around this without transaction. You can force the ordering of the events before transaction, eg. "search .... | sort - _time | transaction ...."

Ledion_Bitincka · ‎05-12-2014

Melonman, what exactly are you trying to get out of the data? Maybe we can find a way to get it without using transactions command

Ledion_Bitincka · ‎05-06-2014

The root cause of this problem is the default setting for vix.yarn.application.classpath in /opt/hunk/etc/system/default/indexes.conf [provider-family:hadoop] .... vix.yarn.application.classpath = $HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,$YARN_HOME/*,$YARN_HOME/lib/* This setting is correct in other Hadoop distros, however the EMR distro seems to use a different value for this setting - you can determine the correct value for it by looking for the value of yarn.application.classpath in the running config of your cluster, http://<resource-manager-host>:9026/conf - in my case, setting vix.yarn.application.classpath as follows got things working!! [provider:emr2] ... vix.yarn.application.classpath = $HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/share/hadoop/common/*,$HADOOP_COMMON_HOME/share/hadoop/common/lib/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,$HADOOP_YARN_HOME/share/hadoop/yarn/*,$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*,/usr/share/aws/emr/emr-fs/lib/*,/usr/share/aws/emr/lib/* another possible solution that might work in your env would be to unset the yarn.application.classpath for your provider and let the default be used [provider:emr2] ... vix.yarn.application.classpath =

Ledion_Bitincka · ‎05-06-2014

I've setup Hunk (6.1) against EMR v2, see contents of indexes.conf below. Streaming searches work perfectly and I can see data come back to Hunk, however as soon as I try to run a reporting search, e.g. "index=emr2 | stats count" I keep getting the following error: [emr2] IOException - Error while waiting for MapReduce job to complete, job_id=[!http://ip-10-34-139-234.ec2.internal:9026/cluster/app/application_1399415419726_0005 job_1399415419726_0005], state=FAILED, reason=Application application_1399415419726_0005 failed 2 times due to AM Container for appattempt_1399415419726_0005_000002 exited with exitCode: 1 due to: Exception from container-launch: and the container logs look like this: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/service/CompositeService at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.service.CompositeService at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 13 more indexes.conf [provider:emr2] vix.command.arg.3 = $SPLUNK_HOME/bin/jars/SplunkMR-s6.0-hy2.0.jar vix.env.HADOOP_HOME = /opt/hadoop/apache/hadoop-2.2.0/ vix.env.JAVA_HOME = /usr/ vix.family = hadoop vix.fs.default.name = hdfs://10.34.139.234:9000/ vix.mapreduce.framework.name = yarn vix.splunk.home.hdfs = /hunk/workdir vix.yarn.resourcemanager.address = 10.34.139.234:9022 vix.yarn.resourcemanager.scheduler.address = 10.34.139.234:9024 [emr2] vix.input.1.path = /hunk/data/... vix.provider = emr2

Ledion_Bitincka · ‎04-29-2014

There are definitely a few optimization knobs that can be turned from the Hunk side - can you please share search.log with us again so we can see what's taking up the time?

Ledion_Bitincka · ‎04-28-2014

There are some known issues with Hunk and Amazon's Hadoop 2.2.0 that cause Hunk to not be able to spawn MapReduce jobs. If you're using the EMR cluster exclusively for Hunk I would recommend that you try Amazon's 1.0.3 version. Also, from the search.log I noticed that there's a possible misconfiguration for the time extraction from the paths, there's a ton of these warning lines: 04-28-2014 19:29:22.819 WARN ERP.elb_log_provider - VixTimeSpecifier - Could not match time regex="/AWSLogs/*/elasticloadbalancing/*/(\d+)/(\d+)/(\d+)/" against path="/logs/AWSLogs/<redacted>" replacing '*' with '.*?' should fix this issue

Ledion_Bitincka · ‎04-28-2014

Thanks for your honest feedback! We'll certainly look into improving the first time run experience. We always welcome any suggestions you might have.

Ledion_Bitincka · ‎04-25-2014

Can you please share the full contents of search.log with us? As well as what version of EMR are you trying to use?

Ledion_Bitincka · ‎04-25-2014

This seems to be like a Hadoop client library version mismatch - the version numbers reported in the error message indicate that the EMR cluster is running Hadoop 2.x while Hunk is trying to use Hadoop 1.x client libraries to communicate with it. Can you please first confirm the EMR version and Hadoop client library version in Hunk before proceeding any further?

Posts	313
Solutions	72
Karma Given	53
Karma Received	252
Member Since	‎01-15-2010

Online Status	Offline
Date Last Visited	‎06-05-2020 02:02 AM

How to get Hunk to recognize timestamps with sub-s...

Hunk has problems running MapReduce jobs against E...

ERROR while waiting for MapReduce job to complete ...

Failed to start MapReduce job. Please consult sear...

EMR issues, MapReduce job killed

Hunk search problem: Cannot initialize Cluster.

Issue reading snappy files using Hunk

Exception when running a search in Hunk

Error while running Hunk search: Cannot initialize...

what does connected mean in transaction

Re: Running Hunk against EMR/S3 without using the ...

Re: Are there hooks into an ERP app when a job is ...

Re: parse fields with multiple delimiters

Re: Hunk Job get OutOfMemory Error

Re: Max of Distinct Count

Re: Hunk Job get OutOfMemory Error

Re: I would like to use compression when submittin...

Re: Is Hunk HDInsight compatible?

Re: hunk installation

Re: Virtual Index with Hunk

Re: Creating Hunk 6.1 Virtual Index

Re: Does Hunk create an index?

Re: Hunk Jobs not getting finalized Yarn HDP 2.x

Re: Hunk Jobs not getting finalized Yarn HDP 2.x

Re: Hunk Jobs not getting finalized Yarn HDP 2.x

Re: Hunk Jobs not getting finalized Yarn HDP 2.x

Re: Getting transaction without using transaction ...

Re: Getting transaction without using transaction ...

Re: Hunk has problems running MapReduce jobs again...

Hunk has problems running MapReduce jobs against E...

Re: Null pointer with Hunk and EMR

Re: Null pointer with Hunk and EMR

Re: Hunk App for AWS ELB - Setup

Re: Null pointer with Hunk and EMR

Re: Hunk App for AWS ELB - Setup

Join the Conversation