Splunk App For Infrastructure: How to get metrics from the actual Splunk Servers

[myfriendhenry]

I have a fully distributed Splunk with 2 searchheads and 2 indexers w/Master.
All of my other Linux hosts are sending the collectd data to the locally installed Universal Forwarder, this works great. It is useful because all the forwarders get the indexer IP's by inquiring from the Master.
I followed this topic:
https://docs.splunk.com/Documentation/InfraApp/1.3.0/Admin/WriteCollectdToUF

Now, I am confused how to configure this on the SearchHeads, Indexers and Master. I want to monitor metrics for those systems as well. Can I use the same guide? I did try a few things, but was not sure where to put the inputs.config. Is this even possible?

[myfriendhenry]

OK, got BOTH SH's sending metrics to ONE SH, the other SH only gets it's OWN metrics - thinking network issue.

Unable to get ANY metrics out of the MASTER however.

[evania]

Hi @myfriendhenry ,

Did you have a chance to check out an answer? If it worked, please resolve this post by approving it! If your problem is still not solved, keep us updated so that someone else can help you.

Thanks for posting!

[ntankersley_spl]

You should not run the script on the the SH, Master or INdexers as it can conflict with inputs and configs on those entities. Please follow the "Manually configure metrics collection on a *nix host" section of docs for setting up collectd on these nodes

https://docs.splunk.com/Documentation/InfraApp/1.3.0/Admin/ManageAgents

[myfriendhenry]

my boss advised the following:
So you will need to install the Add-on and the inputs.conf file onto the Splunk Enterprise server itself under /opt/splunk/etc/apps/Add-on/local/inputs.conf. which should match the inputs.conf that you have deployed out to the forwarders under /opt/splunkforwarder/etc/apps/Add-on/local/inputs.conf. And the data should start being collected. If the Splunk Enterprise server is not the indexer, make sure that you have your outputs.conf configured on the server under /opt/splunk/etc/system/local/outputs.conf which should match your forwarders under /opt/splunkforwarder/etc/system/local/outputs.conf, unless you specify it under another add-on.

[dagarwal_splunk]

I think this might work:
1. In SH and Indexers you will have collectd running (pointing to "localhost").
2. SH collectd data forwarded to Indexer using inputs.conf (udp input) and outputs.conf (https://answers.splunk.com/answers/4209/search-head-configured-as-a-forwarder.html) OR install a new UF ?
3. Indexer you add inputs.conf (udp input). You should already have the SAI Add-on here.

[ntankersley_spl]

You can use the config files for collectd on your monitored hosts, look under /etc/collectd/collectd.conf (Most Linux and Unix) or /etc/collectd.conf (RHEL)

[myfriendhenry]

I understand the collectd part, np. I use the manual method and open a udp port as in "https://docs.splunk.com/Documentation/InfraApp/1.3.0/Admin/WriteCollectdToUF" and it works perfectly for my universal forwarders.
So I got all of that.
My confusion is WHERE to put INPUTS.CONF on the SEARCHHEADS and INDEXERS to monitor THEM with the app.