All Apps and Add-ons

Error in 'SearchParser': Hadoop Ops App

manus2
New Member

Hi,

We are getting this error after installing "Hadoop ops" app.

Error in 'SearchParser': Could not find macro 'hadoop_mr_summary_table' that takes 0 arguments. Expecting stanza name 'hadoop_mr_summary_table'.
Error in 'SearchParser': Could not find macro 'hadoop_task_summary_table' that takes 0 arguments. Expecting stanza name 'hadoop_task_summary_table'.
Error in 'SearchParser': Could not find macro 'hadoop_job_summary_table' that takes 0 arguments. Expecting stanza name 'hadoop_job_summary_table'.

We had installed splunk_for_hadoopops & Splunk_TA_hadoopops in /opt/splunk/etc/apps

We had added /opt/splunk/etc/apps/splunk_for_hadoopops/local/inputs.conf with the help of inputs.conf.cdh3.example.

Edited /opt/splunk/etc/apps/Splunk_TA_hadoopops/default/eventgen.conf with correct and splunk port and password.

But still we are getting the error.

Any thoughts?

Thanks

0 Karma
1 Solution

lyuanlai_splunk
Splunk Employee
Splunk Employee

a quickfix worth trying is running this search query in 'HadoopOps'->'Search'

"|savedsearch __generate_lookup_hadoop_host2hdfs"

You should see a message like "Results written to file '/opt/splunk/etc/apps/SA-HadoopOps/lookups/hadoop_host2hdfs.csv'". If this works, same trick for host2mapred:

"|savedsearch __generate_lookup_hadoop_host2mapred"

View solution in original post

lyuanlai_splunk
Splunk Employee
Splunk Employee

a quickfix worth trying is running this search query in 'HadoopOps'->'Search'

"|savedsearch __generate_lookup_hadoop_host2hdfs"

You should see a message like "Results written to file '/opt/splunk/etc/apps/SA-HadoopOps/lookups/hadoop_host2hdfs.csv'". If this works, same trick for host2mapred:

"|savedsearch __generate_lookup_hadoop_host2mapred"

lyuanlai_splunk
Splunk Employee
Splunk Employee

Please manually set "disabled = 0" for hadoopmon_cpu.sh and restart splunkd. Hopefully in about 5 minutes hadoop_host2maxcpu lookup would be generated

0 Karma

manus2
New Member
  1. Getting CPU events
  2. Getting cpu related o/p
  3. Not enabled ./hopsconfig.sh --auth admin:pass@123 --enable-all throwing error

enabling /opt/splunk/etc/apps/Splunk_TA_hadoopops/bin/hadoopmon_cpu.sh
enable failed

enabling /opt/splunk/etc/apps/Splunk_TA_hadoopops/bin/hadoopmon_df.sh
enable failed

enabling /opt/splunk/etc/apps/Splunk_TA_hadoopops/bin/hadoopmon_dfsreport.sh
enable failed

0 Karma

lyuanlai_splunk
Splunk Employee
Splunk Employee

This one depends on the scripted input 'hadoopmon_cpu.sh' be invoked and run correctly.

1) check if 'source=cpu' returns events
2) check if executing the script 'Splunk_TA_hadoopops/bin/hadoopmon_cpu.sh' in a terminal returns data like

CPU pctUser pctNice pctSystem pctIowait pctIdle
all 1.52 0.00 2.02 0.00 96.46
0 1.01 0.00 2.02 0.00 96.97
1 2.00 0.00 2.00 0.00 96.00

3) check with 'hopsconfig.sh --list-all' whether hadoopmon_cpu.sh is enabled

0 Karma

manus2
New Member

I manually indexed the mapred-site.xml

When running '|savedsearch __generate_lookup_hadoop_host2mapred' we are getting:
"Results written to file '/opt/splunk/etc/apps/SA-HadoopOps/lookups/hadoop_host2mapred.csv"

Now it is showing
Lookup table 'hadoop_host2maxcpu' is empty.

0 Karma

lyuanlai_splunk
Splunk Employee
Splunk Employee

I suspect the mapred-site.xml that is needed to create the hadoop_host2mapred lookup is not being indexed by Splunk. The best advice I can give right now without spamming this thread anymore is to verify that mapred-site.xml is being indexed by Splunk.

0 Karma

manus2
New Member
  1. it was generated automatically and I restarted splunk
  2. same error when running '|savedsearch __generate_lookup_hadoop_host2mapred'

No results. Created empty file 'hadoop_host2mapred'
No matching fields exist

0 Karma

lyuanlai_splunk
Splunk Employee
Splunk Employee

The 3 enabling failures were probably caused by those files/directory not existed yet.

Now with local/inputs.conf generated,
1) do you see monitor:///usr/lib/hadoop/conf/*.xml enabled? If not, please manually add the corresponding entry (suitable for your cluster) to local/inputs.conf and restart splunkd. (see the example above)

2) is hadoop_host2mapred lookup still empty? if so, does running '|savedsearch __generate_lookup_hadoop_host2mapred' fix the problem?

0 Karma

manus2
New Member
  1. deleted eventgen.conf
  2. tried step 3 mentioned in that link by deleting .introspect file & inputs.conf
  3. .introspect file has been created after restarting splunk
  4. but ./hopsconfig.sh --auth : --enable-all gives output like: enabling /usr/lib/hadoop/logs/hadoop*jobsummary* enable failed

enabling /usr/lib/hadoop-0.20/logs/hadoop-hdfs-datanode- user-ThinkCentre-M70e*
enable failed

enabling /usr/lib/hadoop-0.20/logs/hadoop-hdfs-secondarynamenode-user-ThinkCentre-M70e*
enable failed

  1. I did restart whenever I modify inputs.conf
0 Karma

lyuanlai_splunk
Splunk Employee
Splunk Employee

1) You can safely delete eventgen.conf. It is not used.
2) Can you try the steps here http://docs.splunk.com/Documentation/HadoopOps/latest/HadoopOps/DeployandlaunchTA#Update_forwarders_... and see if it fixes the problem?
3) To be cautious... did you restart splunkd after making changes to inputs.conf?

0 Karma

manus2
New Member

There is some stanza related issue is coming up in every restart of splunk after installing this app.

            Possible typo in stanza [sample.jmx.tasktracker.Hadoop] in /opt/splunk/etc/apps/Splunk_TA_hadoopops/default/eventgen.conf, line 40: splunkPort  =  8989
            Possible typo in stanza [sample.jmx.tasktracker.Hadoop] in /opt/splunk/etc/apps/Splunk_TA_hadoopops/default/eventgen.conf, line 42: splunkPass  =  $p1unK_Lab

I retried after changing the splunk port & password with 8089 and splunk password. But it remains same.

0 Karma

manus2
New Member

I added

[monitor:///etc/hadoop/conf/*.xml]
crcSalt =
disabled = 0
sourcetype = hadoop_global_conf
index = hadoopmon_configs

in /opt/splunk/etc/apps/Splunk_TA_hadoopops/local/inputs.conf

But no luck.

0 Karma

lyuanlai_splunk
Splunk Employee
Splunk Employee

For reference, below is a working example of local/inputs.conf of CDH3 namenode.

[monitor:///usr/lib/hadoop/conf/*.xml]
crcSalt =
disabled = 0
sourcetype = hadoop_global_conf
index = hadoopmon_configs

[monitor:///var/log/hadoop/hadoop-cmf-hdfs1-NAMENODE-cdh1.tw.splunk.com*.out]
disabled = 0
sourcetype = hadoop_namenode
index = hadoopmon_logs

0 Karma

lyuanlai_splunk
Splunk Employee
Splunk Employee

I should make a correction here, check if xml files in your HADOOP_CONF_DIR is being monitored in Splunk_TA_hadoopops/local/inputs.conf, something like

[monitor:///etc/hadoop/conf/*.xml]
crcSalt =
disabled = 0
sourcetype = hadoop_global_conf
index = hadoopmon_configs

However, I just remembered you have host2hdfs run correctly, which means it should be there.

0 Karma

manus2
New Member

I am reinstalling the app. Copied "SA-HadoopOps, splunk_for_hadoopops & Splunk_TA_hadoopops" to /opt/splunk/etc/apps/ and restarted the splunk.

./Splunk_TA_hadoopops/default/inputs.conf
./SA-HadoopOps/default/inputs.conf

in which inputs.conf we need to add monitoring lines for "mapred-site.xml"?

0 Karma

lyuanlai_splunk
Splunk Employee
Splunk Employee

check if mapred-site.xml is being monitored, if not, add it then re-run the savedsearch

0 Karma

manus2
New Member

There is no results

0 Karma

lyuanlai_splunk
Splunk Employee
Splunk Employee

Run this search (enclose with backquotes, this is a macro):
"hadoop_mapred_config"

Do you see any results?

0 Karma

manus2
New Member

saved search "__generate_lookup_hadoop_host2hdfs" gives a msg "Results written to file '/opt/splunk/etc/apps/SA-HadoopOps/lookups/hadoop_host2hdfs.csv'"

But saved search "__generate_lookup_hadoop_host2mapred" results:
"No results. Created empty file 'hadoop_host2mapred'
No matching fields exist"

0 Karma

manus2
New Member

Hi lyuanlai,

Thank you for your quick responses.

Now we are getting different error.

"Lookup table 'hadoop_host2maxcpu' is empty.
No matching fields exist
Lookup table 'hadoop_host2mapred' is empty.
No matching fields exist"

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...