topic Anyone splunking hadoop logs? in Getting Data In

Anyone splunking hadoop logs?

pde — Wed, 06 Oct 2010 01:02:33 GMT

I'd be interested in talking about saved searches, etc around hadoop logs. Anybody got a head start?

Thanks!

Re: Anyone splunking hadoop logs?

Dan — Wed, 06 Oct 2010 01:13:53 GMT

Is this for monitoring and troubleshooting Hadoop jobs?

Re: Anyone splunking hadoop logs?

pde — Wed, 06 Oct 2010 01:31:09 GMT

For monitoring & troubleshooting mapreduce jobs and for monitoring/troubleshooting of hadoop clusters overall. My thinking is around a Hadoop app that would compliment and augment hadoop/ganglia.

Re: Anyone splunking hadoop logs?

Archana — Sat, 09 Oct 2010 12:44:40 GMT

I did some performance modeling based on data from hadoop logs and have recently gotten up to speed writing saved searches in Splunk. I'd be happy to help you put the two together.

Re: Anyone splunking hadoop logs?

smcavoyams — Wed, 15 Jun 2011 14:27:56 GMT

Could anyone post saved searches useful field extraction regexs, etc. for hadoop/hbase/etc. ?

Re: Anyone splunking hadoop logs?

Damien_Dallimor — Sun, 17 Jul 2011 03:49:37 GMT

Have a look at Splunk for JMX... Hadoop has loads of Mbeans that you can monitor across your cluster(s).
Many folks are already using this Splunk app for Hadoop and Cassandra monitoring.

Damien.

Re: Anyone splunking hadoop logs?

pierre4splunk — Sun, 29 Jul 2012 00:02:03 GMT

here at splunk R&D, we've been splunking Hadoop logs for awhile: daemon logs, counters and measures... metrics, metrics, metrics. things get very interesting when you're able to correlate this with system info for each node in the cluster, client activity, configuration settings, and more. we've developed an app that collects all this for splunk search, along with rich dashboards and interactive UIs designed for both Hadoop operators and developers' needs.

interested in trying it out? we'd love to share ideas and get your feedback.

To be eligible, register for the splunk enterprise for Hadoop beta program on splunkbase. Be sure to describe what you're interested in too -- i.e. mention 'monitor' or 'splunking Hadoop logs' in the web form.

Re: Anyone splunking hadoop logs?

pierre4splunk — Sat, 27 Oct 2012 23:47:36 GMT

The Splunk App for HadoopOps is now generally available.

http://splunk-base.splunk.com/apps/57004/splunk-app-for-hadoopops

The App is free. You can learn more here:

http://www.splunk.com/view/hadoopops/SP-CAAAHA2