All Apps and Add-ons

Continuous monitoring using hadoop connect app

Explorer

Hi All,

I am trying to monitor HDFS files in splunk using hadoop connect app, HDFS directory structure changes everyday and splunk hadoop connect is not able to pull in new files into splunk. wondering if this can be achieved or I am missing something here ?

0 Karma

Splunk Employee
Splunk Employee

Using the Hadoop Connect Import function, make sure that the Resource name path is high level enough to deal with the daily changes. So if for example you have hadoop.example.com:8020/year/month/day/hour/ you can change it to hadoop.example.com:8020/year/
https://docs.splunk.com/Documentation/HadoopConnect/latest/DeployHadoopConnect/ImportfromHDFS

Also, you may want to look at Splunk Analytics for Hadoop as a way to monitor HDFS files
https://www.splunk.com/blog/2013/11/08/hunk-intro-part-3.html
https://www.splunk.com/blog/2014/05/14/hunkonhunk.html

0 Karma

Influencer
0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!