All Apps and Add-ons

Hadoop Monitoring:How to get field extraction on index=hadoopmon_metrics?

ThomasControlwa
Path Finder

Hi @ all,
we'll test the Hadoop Monitoring APP.
- installation is complete, got events in 2 index, like hadoopmon_os & hadoopmon_metrics.
- the index hadoopmon_os seams to work correctly (shows "interested fields")
- the index hadoopmon_metrics got RAW events but without interested fields

samle of inputsconf of the Forwarder installation
for index hadoopmon_metrics
hadoopmon_metrics

# [monitor:///hadoop/logs/hadoop/hdfs/hadoop-hdfs-namenode*.log]
# index = hadoopmon_metrics
# sourcetype = hadoop_namenode
# disabled = 0
# [monitor:///hadoop/logs/hadoop/hdfs/hadoop-hdfs-namenode*.out]
# index = hadoopmon_metrics
# sourcetype = hadoop_namenode
# disabled = 0

Scripted inputs for index hadoopmon_os works fine
Has someone an idea to got fields / extraction of RAW data?

thanks in advance

0 Karma

mayurr98
Super Champion

if you are using app then it is in opt/splunk/etc/apps/maprops/default
Well, I Installed Hadoop monitoring app on my local system, and there are no field extractions for the mentioned sourcetypes. So you need to extract it manually. There are field extractions only for OS scripted inputs.

let me know if this helps!

ThomasControlwa
Path Finder

I downvoted this post because not helpful, because it doesn't make sense when i'm looking for using the preinstalled frondend. there searches like

[yarn top user]

index=hadoopmon_metrics sourcetype=hadoop_resourcemanager appid=*| top limit=20 user

[yarn success rate]

index=hadoopmon_metrics sourcetype=hadoop_historyserver user=* | eval elapsedtime = finishtime - submittime| table jobname queue user nummaps numreduces status elapsedtime

etc

0 Karma

mayurr98
Super Champion

You are looking for a saved search that you will find in /opt/splunk/splunk/etc/apps/maprops/default/savedsearch.conf
in which they have defined display.events.fields = ["host","source","sourcetype","APPID","CONTAINERID","OPERATION","RESULT","USER","TARGET"]

And you should not downvote the post unless it harms your system.
https://answers.splunk.com/answers/244111/proper-etiquette-and-timing-for-voting-here-on-ans.html

0 Karma

mayurr98
Super Champion

Also, does your raw data contains key-value pair i.e. user=value? if it does then Splunk schema automatically extracts key-value pairs.

0 Karma

ThomasControlwa
Path Finder

thanks for the TIP,
so I agree that the saved search is there,
but why the following line doesn't work?

index=hadoopmon_metrics sourcetype=hadoop_resourcemanager appId=* | top finalStatus

when I' looking for "index=hadoopmon_metrics sourcetype=hadoop_resourcemanager" there no field extractions just like "appId"

do you have data in your test inv?
cheers, and so sry for downvoting

0 Karma

mayurr98
Super Champion

No, I do not have sample data for this, are you running index=hadoopmon_metrics sourcetype=hadoop_resourcemanager in verbose mode?
I am quite sure you are running it in fast mode where you mostly will not see all the fields.
Below time picker can you see three modes? verbose mode will give you all the fields.

0 Karma

ThomasControlwa
Path Finder

yes runs in verbose mode, and use environment data of hadoop (just a other team)

0 Karma

ThomasControlwa
Path Finder

you are right, there no data with this key-value pair i.e. user=value

many thanks for your support!

0 Karma

ThomasControlwa
Path Finder

where i can find the props.conf for sourcetype like:
# hadoop_datanode
# hadoop_namenode
# hadoop_historyserver

0 Karma
Get Updates on the Splunk Community!

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...