The Cloudera specific log location should be here: ### Cloudera Yarn Log Files [monitor:///var/log/hadoop-yarn/*nodemanager*] sourcetype = hadoop_nodemanager index = hadoopmon_metrics [monitor:///var/log/hadoop-yarn/*resourcemanager*] sourcetype = hadoop_resourcemanager index = hadoopmon_metrics [monitor:///var/log/hadoop-yarn/*proxyserver*] sourcetype = hadoop_proxyserver index = hadoopmon_metrics [monitor:///var/log/hadoop-mapreduce/*historyserver*] sourcetype = hadoop_historyserver index = hadoopmon_metric ### Cloudera Hadoop Log Files [monitor:///var/log/hadoop-hdfs/*datanode*] sourcetype = hadoop_datanode index = hadoopmon_metrics [monitor:///var/log/hadoop-hdfs/*namenode*] sourcetype = hadoop_namenode index = hadoopmon_metrics [monitor:///var/log/hadoop-hdfs/*secondarynamenode*] sourcetype = hadoop_secndarynamenode index = hadoopmon_metrics [monitor:///var/log/hadoop-hdfs/*journalnode*] sourcetype = hadoop_journalnode index = hadoopmon_metrics ### Cloudera Configuration Files [monitor:///etc/hadoop/conf/*] crcSalt = <SOURCE> disabled = 0 sourcetype = hadoop_global_conf index = hadoopmon_configs And after you collect the logs you can run searches similar to these: [Yarn All Applications] index=hadoopmon_metrics sourcetype=hadoop_resourcemanager appId=* | eval elapsed_time = finishTime - startTime | table appId name user queue finalStatus elapsed_time [Yarn Top User] index=hadoopmon_metrics sourcetype=hadoop_resourcemanager appId=* | top user [Yarn Success Rate] index=hadoopmon_metrics sourcetype=hadoop_resourcemanager appId=* | top finalStatus ... View more

Hi, yes I got it to work. Try: Name Node = maprfs:///  Since MapR use Yarn the important flags are  vix.yarn.resourcemanager.address vix.yarn.resourcemanager.scheduler.address ... View more

Hi, these solutions actually used Splunk DB Connect with JDBC to talk to Spark SQL ... View more

These two flags are misconfigured. it should be masternode:8030 not hdfs://masternode:8030 and same for the 8032 ... View more

Can you see if these two Hadoop flags: yarn.resourcemanager.address (normally port 8032) / yarn.resourcemanager.scheduler.address (normally port 8030) are valid and have the same values you see in the Splunk Provider ? ... View more

splunk analysis for hadoop is already included as part of Splunk Enterprise. You will get 60 days license once you download Splunk. The above add-on is just to request an additional license past the initial 60 days. ... View more

Let me know and we can try to debug these issues? My email is rdagan@splunk.com ... View more

Splunk Hadoop Data Roll can backup your indexes into HDFS or S3 (we recommend that you use S3A and not S3). Here are few links that can help: https://docs.splunk.com/Documentation/Splunk/latest/Indexer/ArchivingindexestoHadoop https://docs.splunk.com/Documentation/Splunk/7.2.1/Indexer/ArchivingSplunkindexestoS3 https://answers.splunk.com/search.html?f=&redirect=search%2Fsearch&sort=relevance&q=s3a&type=question https://www.splunk.com/blog/2015/02/11/faster-and-limitless-hunk-archiving-to-s3-with-hadoop-2-6-0.html ... View more

Can I assume that majority of the buckets made it to S3, but the above 5 did not? Are you using S3A or S3 in the VIX? S3A does not have size limitation, so I am trying to eliminate that as the cause ... View more

It looks as if these buckets that have errors are going back 10 months (November 27, 2018). Can you check if the data is still in Splunk? If it is, can you check if these buckets made it to S3 despite of the exception? ... View more

The recommended way to read data from Hadoop to Splunk is to use Splunk Analytics for Hadoop: https://docs.splunk.com/Documentation/Splunk/latest/HadoopAnalytics/MeetSplunkAnalyticsforHadoop The recommended way to write data from Splunk to Hadoop is to use Splunk Hadoop Data Roll: https://docs.splunk.com/Documentation/Splunk/latest/Indexer/ArchivingindexestoHadoop If you want to use Presto, our recommendation is to use Splunk DB Connect (https://docs.splunk.com/Documentation/DBX) with Presto JDBC driver (for example, the JDBC found here https://prestodb.github.io/docs/current/installation/jdbc.html ) ... View more

Let me know and we can schedule a webex to debug these issues? My Splunk email is rdagan ... View more

Can you verify that this flag vix.input.1.path = /tmp/splunk/*/... is pointing to HDFS (for example, /user/splunk/data/mylogs/...)? It looks as if you are using the same local directory path as your vix.splunk.home.datanode = /tmp/splunk/$SPLUNK_SERVER_NAME/ That will explain the mix results. ... View more

By default I see ' user ' + 3 capabilities [root@localhost default]# pwd /opt/splunk/etc/apps/HadoopConnect/default [root@localhost default]# more authorize.conf [capability::hc_read] [capability::hc_write] [role_hc_admin] importRoles = user hc_read = enabled hc_write = enabled schedule_search = enabled ... View more

As you noticed Hadoop Connect export can only use index time and not event time. Therefore, I would recommend you modify your search to try and use the flags _index_earliest and _index_latest For example, index=data_index_to_export _index_earliest=-7h@h _index_latest=-6h@h This link maybe helpful: https://docs.splunk.com/Documentation/Splunk/7.2.4/SearchReference/Searchtimemodifiers ... View more

As you can see from this video ( https://www.youtube.com/watch?v=TmYHsabpk_Q ) When you setup the Java path and Hadoop path inside Splunk Hadoop Connect, you should not include the ' bin/hadoop or bin/java part Can you share the output of these ' which hadoop ' and ' which java ' commands? ... View more

How about the product Splunk Analytics for Hadoop? Is that product also has org restrictions? https://www.splunk.com/en_us/products/apps-and-add-ons/splunk-analytics-for-hadoop.html ... View more

To debug this, from the command line, can you run the commands: which hadoop and which java ... View more

Using the Hadoop Connect Import function, make sure that the Resource name path is high level enough to deal with the daily changes. So if for example you have hadoop.example.com:8020/year/month/day/hour/ you can change it to hadoop.example.com:8020/year/ https://docs.splunk.com/Documentation/HadoopConnect/latest/DeployHadoopConnect/ImportfromHDFS Also, you may want to look at Splunk Analytics for Hadoop as a way to monitor HDFS files https://www.splunk.com/blog/2013/11/08/hunk-intro-part-3.html https://www.splunk.com/blog/2014/05/14/hunkonhunk.html ... View more

If your license is Splunk Analytics for Hadoop (Hunk), the UI will show Hunk. ... View more

For this type of use case (Hadoop directory usage), I would suggest you use Splunk Hadoop Connect App (not Splunk Analytics for Hadoop - Hunk). Specifically, try the command " | hdfs lsr " Here is a good blog on the subject: https://www.splunk.com/blog/2012/12/20/connecting-splunk-and-hadoop.html ... View more

Do you see any errors? ... View more

It sounds like the right App is actually the Hadoop Monitoring: Hadoop monitoring – Read Hadoop logs and monitor Hadoop health and jobs ( https://splunkbase.splunk.com/app/3134/ ) Hadoop Data Roll - sends raw data from Splunk buckets to Hadoop Splunk Analytics for Hadoop (Hunk) - Generate MapReduce Jobs to read data from HDFS Splunk Hadoop Connect Export - Send results from Splunk SH to HDFS Splunk Hadoop Connect Import - Read files from HDFS and Index them in Splunk ... View more

The process should be something along the lines of: 1) package your local app tar cvzf my_hadoopconnect.tgz /splunk/etc/apps/HadoopConnect 2) untar in your dev tar xvzf my_hadoopconnect.tgz -C /splunk/etc/apps/HadoopConnect 3) restart splunk /splunk/bin/splunk restart Here are the steps: http://docs.splunk.com/Documentation/HadoopConnect/1.2.5/DeployHadoopConnect/Downloadandinstalltheapp#Install_the_app_on_a_full_Splunk_instance ... View more