Anyone splunking hadoop logs?

[pde]

I'd be interested in talking about saved searches, etc around hadoop logs. Anybody got a head start?

Thanks!

[pierre4splunk]

The Splunk App for HadoopOps is now generally available.

http://splunk-base.splunk.com/apps/57004/splunk-app-for-hadoopops

The App is free. You can learn more here:

http://www.splunk.com/view/hadoopops/SP-CAAAHA2

[pierre4splunk]

here at splunk R&D, we've been splunking Hadoop logs for awhile: daemon logs, counters and measures... metrics, metrics, metrics. things get very interesting when you're able to correlate this with system info for each node in the cluster, client activity, configuration settings, and more. we've developed an app that collects all this for splunk search, along with rich dashboards and interactive UIs designed for both Hadoop operators and developers' needs.

interested in trying it out? we'd love to share ideas and get your feedback.

To be eligible, register for the splunk enterprise for Hadoop beta program on splunkbase. Be sure to describe what you're interested in too -- i.e. mention 'monitor' or 'splunking Hadoop logs' in the web form.

[Damien_Dallimor]

Have a look at Splunk for JMX... Hadoop has loads of Mbeans that you can monitor across your cluster(s).
Many folks are already using this Splunk app for Hadoop and Cassandra monitoring.

Damien.

[smcavoyams]

Could anyone post saved searches useful field extraction regexs, etc. for hadoop/hbase/etc. ?

[Archana]

I did some performance modeling based on data from hadoop logs and have recently gotten up to speed writing saved searches in Splunk. I'd be happy to help you put the two together.

[pde]

For monitoring & troubleshooting mapreduce jobs and for monitoring/troubleshooting of hadoop clusters overall. My thinking is around a Hadoop app that would compliment and augment hadoop/ganglia.

[Dan]

Is this for monitoring and troubleshooting Hadoop jobs?