Hadoop Monitoring:How to get field extraction on index=hadoopmon_metrics?

[ThomasControlwa]

Hi @ all,
we'll test the Hadoop Monitoring APP.
- installation is complete, got events in 2 index, like hadoopmon_os & hadoopmon_metrics.
- the index hadoopmon_os seams to work correctly (shows "interested fields")
- the index hadoopmon_metrics got RAW events but without interested fields

samle of inputsconf of the Forwarder installation
for index hadoopmon_metrics
hadoopmon_metrics

# [monitor:///hadoop/logs/hadoop/hdfs/hadoop-hdfs-namenode*.log]
# index = hadoopmon_metrics
# sourcetype = hadoop_namenode
# disabled = 0
# [monitor:///hadoop/logs/hadoop/hdfs/hadoop-hdfs-namenode*.out]
# index = hadoopmon_metrics
# sourcetype = hadoop_namenode
# disabled = 0

Scripted inputs for index hadoopmon_os works fine
Has someone an idea to got fields / extraction of RAW data?

thanks in advance

[mayurr98]

if you are using app then it is in opt/splunk/etc/apps/maprops/default
Well, I Installed Hadoop monitoring app on my local system, and there are no field extractions for the mentioned sourcetypes. So you need to extract it manually. There are field extractions only for OS scripted inputs.

let me know if this helps!

[ThomasControlwa]

I downvoted this post because not helpful, because it doesn't make sense when i'm looking for using the preinstalled frondend. there searches like

[yarn top user]

index=hadoopmon_metrics sourcetype=hadoop_resourcemanager appid=*| top limit=20 user

[yarn success rate]

index=hadoopmon_metrics sourcetype=hadoop_historyserver user=* | eval elapsedtime = finishtime - submittime| table jobname queue user nummaps numreduces status elapsedtime

etc

[mayurr98]

You are looking for a saved search that you will find in /opt/splunk/splunk/etc/apps/maprops/default/savedsearch.conf
in which they have defined display.events.fields = ["host","source","sourcetype","APPID","CONTAINERID","OPERATION","RESULT","USER","TARGET"]

And you should not downvote the post unless it harms your system.
https://answers.splunk.com/answers/244111/proper-etiquette-and-timing-for-voting-here-on-ans.html

[mayurr98]

Also, does your raw data contains key-value pair i.e. user=value? if it does then Splunk schema automatically extracts key-value pairs.

[ThomasControlwa]

thanks for the TIP,
so I agree that the saved search is there,
but why the following line doesn't work?

index=hadoopmon_metrics sourcetype=hadoop_resourcemanager appId=* | top finalStatus

when I' looking for "index=hadoopmon_metrics sourcetype=hadoop_resourcemanager" there no field extractions just like "appId"

do you have data in your test inv?
cheers, and so sry for downvoting

[mayurr98]

No, I do not have sample data for this, are you running index=hadoopmon_metrics sourcetype=hadoop_resourcemanager in verbose mode?
I am quite sure you are running it in fast mode where you mostly will not see all the fields.
Below time picker can you see three modes? verbose mode will give you all the fields.

[ThomasControlwa]

yes runs in verbose mode, and use environment data of hadoop (just a other team)

[ThomasControlwa]

you are right, there no data with this key-value pair i.e. user=value

many thanks for your support!

[ThomasControlwa]

where i can find the props.conf for sourcetype like:
# hadoop_datanode
# hadoop_namenode
# hadoop_historyserver