All Apps and Add-ons

Hadoop Monitoring:How to get field extraction on index=hadoopmon_metrics?

ThomasControlwa
Path Finder

Hi @ all,
we'll test the Hadoop Monitoring APP.
- installation is complete, got events in 2 index, like hadoopmon_os & hadoopmon_metrics.
- the index hadoopmon_os seams to work correctly (shows "interested fields")
- the index hadoopmon_metrics got RAW events but without interested fields

samle of inputsconf of the Forwarder installation
for index hadoopmon_metrics
hadoopmon_metrics

# [monitor:///hadoop/logs/hadoop/hdfs/hadoop-hdfs-namenode*.log]
# index = hadoopmon_metrics
# sourcetype = hadoop_namenode
# disabled = 0
# [monitor:///hadoop/logs/hadoop/hdfs/hadoop-hdfs-namenode*.out]
# index = hadoopmon_metrics
# sourcetype = hadoop_namenode
# disabled = 0

Scripted inputs for index hadoopmon_os works fine
Has someone an idea to got fields / extraction of RAW data?

thanks in advance

0 Karma

mayurr98
Super Champion

if you are using app then it is in opt/splunk/etc/apps/maprops/default
Well, I Installed Hadoop monitoring app on my local system, and there are no field extractions for the mentioned sourcetypes. So you need to extract it manually. There are field extractions only for OS scripted inputs.

let me know if this helps!

ThomasControlwa
Path Finder

I downvoted this post because not helpful, because it doesn't make sense when i'm looking for using the preinstalled frondend. there searches like

[yarn top user]

index=hadoopmon_metrics sourcetype=hadoop_resourcemanager appid=*| top limit=20 user

[yarn success rate]

index=hadoopmon_metrics sourcetype=hadoop_historyserver user=* | eval elapsedtime = finishtime - submittime| table jobname queue user nummaps numreduces status elapsedtime

etc

0 Karma

mayurr98
Super Champion

You are looking for a saved search that you will find in /opt/splunk/splunk/etc/apps/maprops/default/savedsearch.conf
in which they have defined display.events.fields = ["host","source","sourcetype","APPID","CONTAINERID","OPERATION","RESULT","USER","TARGET"]

And you should not downvote the post unless it harms your system.
https://answers.splunk.com/answers/244111/proper-etiquette-and-timing-for-voting-here-on-ans.html

0 Karma

mayurr98
Super Champion

Also, does your raw data contains key-value pair i.e. user=value? if it does then Splunk schema automatically extracts key-value pairs.

0 Karma

ThomasControlwa
Path Finder

thanks for the TIP,
so I agree that the saved search is there,
but why the following line doesn't work?

index=hadoopmon_metrics sourcetype=hadoop_resourcemanager appId=* | top finalStatus

when I' looking for "index=hadoopmon_metrics sourcetype=hadoop_resourcemanager" there no field extractions just like "appId"

do you have data in your test inv?
cheers, and so sry for downvoting

0 Karma

mayurr98
Super Champion

No, I do not have sample data for this, are you running index=hadoopmon_metrics sourcetype=hadoop_resourcemanager in verbose mode?
I am quite sure you are running it in fast mode where you mostly will not see all the fields.
Below time picker can you see three modes? verbose mode will give you all the fields.

0 Karma

ThomasControlwa
Path Finder

yes runs in verbose mode, and use environment data of hadoop (just a other team)

0 Karma

ThomasControlwa
Path Finder

you are right, there no data with this key-value pair i.e. user=value

many thanks for your support!

0 Karma

ThomasControlwa
Path Finder

where i can find the props.conf for sourcetype like:
# hadoop_datanode
# hadoop_namenode
# hadoop_historyserver

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...