All Apps and Add-ons

Splunk App For Infrastructure: How to get metrics from the actual Splunk Servers

myfriendhenry
Explorer

I have a fully distributed Splunk with 2 searchheads and 2 indexers w/Master.
All of my other Linux hosts are sending the collectd data to the locally installed Universal Forwarder, this works great. It is useful because all the forwarders get the indexer IP's by inquiring from the Master.
I followed this topic:
https://docs.splunk.com/Documentation/InfraApp/1.3.0/Admin/WriteCollectdToUF

Now, I am confused how to configure this on the SearchHeads, Indexers and Master. I want to monitor metrics for those systems as well. Can I use the same guide? I did try a few things, but was not sure where to put the inputs.config. Is this even possible?

0 Karma

myfriendhenry
Explorer

OK, got BOTH SH's sending metrics to ONE SH, the other SH only gets it's OWN metrics - thinking network issue.

Unable to get ANY metrics out of the MASTER however.

0 Karma

evania
Community Manager
Community Manager

Hi @myfriendhenry ,

Did you have a chance to check out an answer? If it worked, please resolve this post by approving it! If your problem is still not solved, keep us updated so that someone else can help you.

Thanks for posting!

0 Karma

ntankersley_spl
Splunk Employee
Splunk Employee

You should not run the script on the the SH, Master or INdexers as it can conflict with inputs and configs on those entities. Please follow the "Manually configure metrics collection on a *nix host" section of docs for setting up collectd on these nodes

https://docs.splunk.com/Documentation/InfraApp/1.3.0/Admin/ManageAgents

myfriendhenry
Explorer

my boss advised the following:
So you will need to install the Add-on and the inputs.conf file onto the Splunk Enterprise server itself under /opt/splunk/etc/apps/Add-on/local/inputs.conf. which should match the inputs.conf that you have deployed out to the forwarders under /opt/splunkforwarder/etc/apps/Add-on/local/inputs.conf. And the data should start being collected. If the Splunk Enterprise server is not the indexer, make sure that you have your outputs.conf configured on the server under /opt/splunk/etc/system/local/outputs.conf which should match your forwarders under /opt/splunkforwarder/etc/system/local/outputs.conf, unless you specify it under another add-on.

0 Karma

dagarwal_splunk
Splunk Employee
Splunk Employee

I think this might work:
1. In SH and Indexers you will have collectd running (pointing to "localhost").
2. SH collectd data forwarded to Indexer using inputs.conf (udp input) and outputs.conf (https://answers.splunk.com/answers/4209/search-head-configured-as-a-forwarder.html) OR install a new UF ?
3. Indexer you add inputs.conf (udp input). You should already have the SAI Add-on here.

0 Karma

ntankersley_spl
Splunk Employee
Splunk Employee

You can use the config files for collectd on your monitored hosts, look under /etc/collectd/collectd.conf (Most Linux and Unix) or /etc/collectd.conf (RHEL)

0 Karma

myfriendhenry
Explorer

I understand the collectd part, np. I use the manual method and open a udp port as in "https://docs.splunk.com/Documentation/InfraApp/1.3.0/Admin/WriteCollectdToUF" and it works perfectly for my universal forwarders.
So I got all of that.
My confusion is WHERE to put INPUTS.CONF on the SEARCHHEADS and INDEXERS to monitor THEM with the app.

0 Karma
Get Updates on the Splunk Community!

Welcome to the Future of Data Search & Exploration

You have more data coming at you than ever before. Over the next five years, the total amount of digital data ...

What’s new on Splunk Lantern in August

This month’s Splunk Lantern update gives you the low-down on all of the articles we’ve published over the past ...

This Week's Community Digest - Splunk Community Happenings [8.3.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...