All Apps and Add-ons

Continuous monitoring using hadoop connect app

ksolanki88
Explorer

Hi All,

I am trying to monitor HDFS files in splunk using hadoop connect app, HDFS directory structure changes everyday and splunk hadoop connect is not able to pull in new files into splunk. wondering if this can be achieved or I am missing something here ?

0 Karma

rdagan_splunk
Splunk Employee
Splunk Employee

Using the Hadoop Connect Import function, make sure that the Resource name path is high level enough to deal with the daily changes. So if for example you have hadoop.example.com:8020/year/month/day/hour/ you can change it to hadoop.example.com:8020/year/
https://docs.splunk.com/Documentation/HadoopConnect/latest/DeployHadoopConnect/ImportfromHDFS

Also, you may want to look at Splunk Analytics for Hadoop as a way to monitor HDFS files
https://www.splunk.com/blog/2013/11/08/hunk-intro-part-3.html
https://www.splunk.com/blog/2014/05/14/hunkonhunk.html

0 Karma

dkeck
Influencer
0 Karma
Get Updates on the Splunk Community!

Splunk Decoded: Service Maps vs Service Analyzer Tree View vs Flow Maps

It’s Monday morning, and your phone is buzzing with alert escalations – your customer-facing portal is running ...

What’s New in Splunk Observability – September 2025

What's NewWe are excited to announce the latest enhancements to Splunk Observability, designed to help ITOps ...

Fun with Regular Expression - multiples of nine

Fun with Regular Expression - multiples of nineThis challenge was first posted on Slack #regex channel ...