Hi,
We are getting this error after installing "Hadoop ops" app.
Error in 'SearchParser': Could not find macro 'hadoop_mr_summary_table' that takes 0 arguments. Expecting stanza name 'hadoop_mr_summary_table'.
Error in 'SearchParser': Could not find macro 'hadoop_task_summary_table' that takes 0 arguments. Expecting stanza name 'hadoop_task_summary_table'.
Error in 'SearchParser': Could not find macro 'hadoop_job_summary_table' that takes 0 arguments. Expecting stanza name 'hadoop_job_summary_table'.
We had installed splunk_for_hadoopops & Splunk_TA_hadoopops in /opt/splunk/etc/apps
We had added /opt/splunk/etc/apps/splunk_for_hadoopops/local/inputs.conf with the help of inputs.conf.cdh3.example.
Edited /opt/splunk/etc/apps/Splunk_TA_hadoopops/default/eventgen.conf with correct and splunk port and password.
But still we are getting the error.
Any thoughts?
Thanks
a quickfix worth trying is running this search query in 'HadoopOps'->'Search'
"|savedsearch __generate_lookup_hadoop_host2hdfs"
You should see a message like "Results written to file '/opt/splunk/etc/apps/SA-HadoopOps/lookups/hadoop_host2hdfs.csv'". If this works, same trick for host2mapred:
"|savedsearch __generate_lookup_hadoop_host2mapred"
a quickfix worth trying is running this search query in 'HadoopOps'->'Search'
"|savedsearch __generate_lookup_hadoop_host2hdfs"
You should see a message like "Results written to file '/opt/splunk/etc/apps/SA-HadoopOps/lookups/hadoop_host2hdfs.csv'". If this works, same trick for host2mapred:
"|savedsearch __generate_lookup_hadoop_host2mapred"
Please manually set "disabled = 0" for hadoopmon_cpu.sh and restart splunkd. Hopefully in about 5 minutes hadoop_host2maxcpu lookup would be generated
enabling /opt/splunk/etc/apps/Splunk_TA_hadoopops/bin/hadoopmon_cpu.sh
enable failed
enabling /opt/splunk/etc/apps/Splunk_TA_hadoopops/bin/hadoopmon_df.sh
enable failed
enabling /opt/splunk/etc/apps/Splunk_TA_hadoopops/bin/hadoopmon_dfsreport.sh
enable failed
This one depends on the scripted input 'hadoopmon_cpu.sh' be invoked and run correctly.
1) check if 'source=cpu' returns events
2) check if executing the script 'Splunk_TA_hadoopops/bin/hadoopmon_cpu.sh' in a terminal returns data like
CPU pctUser pctNice pctSystem pctIowait pctIdle
all 1.52 0.00 2.02 0.00 96.46
0 1.01 0.00 2.02 0.00 96.97
1 2.00 0.00 2.00 0.00 96.00
3) check with 'hopsconfig.sh --list-all' whether hadoopmon_cpu.sh is enabled
I manually indexed the mapred-site.xml
When running '|savedsearch __generate_lookup_hadoop_host2mapred' we are getting:
"Results written to file '/opt/splunk/etc/apps/SA-HadoopOps/lookups/hadoop_host2mapred.csv"
Now it is showing
Lookup table 'hadoop_host2maxcpu' is empty.
I suspect the mapred-site.xml that is needed to create the hadoop_host2mapred lookup is not being indexed by Splunk. The best advice I can give right now without spamming this thread anymore is to verify that mapred-site.xml is being indexed by Splunk.
No results. Created empty file 'hadoop_host2mapred'
No matching fields exist
The 3 enabling failures were probably caused by those files/directory not existed yet.
Now with local/inputs.conf generated,
1) do you see monitor:///usr/lib/hadoop/conf/*.xml enabled? If not, please manually add the corresponding entry (suitable for your cluster) to local/inputs.conf and restart splunkd. (see the example above)
2) is hadoop_host2mapred lookup still empty? if so, does running '|savedsearch __generate_lookup_hadoop_host2mapred' fix the problem?
enabling /usr/lib/hadoop-0.20/logs/hadoop-hdfs-datanode- user-ThinkCentre-M70e*
enable failed
enabling /usr/lib/hadoop-0.20/logs/hadoop-hdfs-secondarynamenode-user-ThinkCentre-M70e*
enable failed
1) You can safely delete eventgen.conf. It is not used.
2) Can you try the steps here http://docs.splunk.com/Documentation/HadoopOps/latest/HadoopOps/DeployandlaunchTA#Update_forwarders_... and see if it fixes the problem?
3) To be cautious... did you restart splunkd after making changes to inputs.conf?
There is some stanza related issue is coming up in every restart of splunk after installing this app.
Possible typo in stanza [sample.jmx.tasktracker.Hadoop] in /opt/splunk/etc/apps/Splunk_TA_hadoopops/default/eventgen.conf, line 40: splunkPort = 8989
Possible typo in stanza [sample.jmx.tasktracker.Hadoop] in /opt/splunk/etc/apps/Splunk_TA_hadoopops/default/eventgen.conf, line 42: splunkPass = $p1unK_Lab
I retried after changing the splunk port & password with 8089 and splunk password. But it remains same.
I added
[monitor:///etc/hadoop/conf/*.xml]
crcSalt =
in /opt/splunk/etc/apps/Splunk_TA_hadoopops/local/inputs.conf
But no luck.
For reference, below is a working example of local/inputs.conf of CDH3 namenode.
[monitor:///usr/lib/hadoop/conf/*.xml]
crcSalt =
[monitor:///var/log/hadoop/hadoop-cmf-hdfs1-NAMENODE-cdh1.tw.splunk.com*.out]
disabled = 0
sourcetype = hadoop_namenode
index = hadoopmon_logs
I should make a correction here, check if xml files in your HADOOP_CONF_DIR is being monitored in Splunk_TA_hadoopops/local/inputs.conf, something like
[monitor:///etc/hadoop/conf/*.xml]
crcSalt =
However, I just remembered you have host2hdfs run correctly, which means it should be there.
I am reinstalling the app. Copied "SA-HadoopOps, splunk_for_hadoopops & Splunk_TA_hadoopops" to /opt/splunk/etc/apps/ and restarted the splunk.
./Splunk_TA_hadoopops/default/inputs.conf
./SA-HadoopOps/default/inputs.conf
in which inputs.conf we need to add monitoring lines for "mapred-site.xml"?
check if mapred-site.xml is being monitored, if not, add it then re-run the savedsearch
There is no results
Run this search (enclose with backquotes, this is a macro):
"hadoop_mapred_config
"
Do you see any results?
saved search "__generate_lookup_hadoop_host2hdfs" gives a msg "Results written to file '/opt/splunk/etc/apps/SA-HadoopOps/lookups/hadoop_host2hdfs.csv'"
But saved search "__generate_lookup_hadoop_host2mapred" results:
"No results. Created empty file 'hadoop_host2mapred'
No matching fields exist"
Hi lyuanlai,
Thank you for your quick responses.
Now we are getting different error.
"Lookup table 'hadoop_host2maxcpu' is empty.
No matching fields exist
Lookup table 'hadoop_host2mapred' is empty.
No matching fields exist"