All Apps and Add-ons

Splunk App For Infrastructure: How to get metrics from the actual Splunk Servers

myfriendhenry
Explorer

I have a fully distributed Splunk with 2 searchheads and 2 indexers w/Master.
All of my other Linux hosts are sending the collectd data to the locally installed Universal Forwarder, this works great. It is useful because all the forwarders get the indexer IP's by inquiring from the Master.
I followed this topic:
https://docs.splunk.com/Documentation/InfraApp/1.3.0/Admin/WriteCollectdToUF

Now, I am confused how to configure this on the SearchHeads, Indexers and Master. I want to monitor metrics for those systems as well. Can I use the same guide? I did try a few things, but was not sure where to put the inputs.config. Is this even possible?

0 Karma

myfriendhenry
Explorer

OK, got BOTH SH's sending metrics to ONE SH, the other SH only gets it's OWN metrics - thinking network issue.

Unable to get ANY metrics out of the MASTER however.

0 Karma

evania
Splunk Employee
Splunk Employee

Hi @myfriendhenry ,

Did you have a chance to check out an answer? If it worked, please resolve this post by approving it! If your problem is still not solved, keep us updated so that someone else can help you.

Thanks for posting!

0 Karma

ntankersley_spl
Splunk Employee
Splunk Employee

You should not run the script on the the SH, Master or INdexers as it can conflict with inputs and configs on those entities. Please follow the "Manually configure metrics collection on a *nix host" section of docs for setting up collectd on these nodes

https://docs.splunk.com/Documentation/InfraApp/1.3.0/Admin/ManageAgents

myfriendhenry
Explorer

my boss advised the following:
So you will need to install the Add-on and the inputs.conf file onto the Splunk Enterprise server itself under /opt/splunk/etc/apps/Add-on/local/inputs.conf. which should match the inputs.conf that you have deployed out to the forwarders under /opt/splunkforwarder/etc/apps/Add-on/local/inputs.conf. And the data should start being collected. If the Splunk Enterprise server is not the indexer, make sure that you have your outputs.conf configured on the server under /opt/splunk/etc/system/local/outputs.conf which should match your forwarders under /opt/splunkforwarder/etc/system/local/outputs.conf, unless you specify it under another add-on.

0 Karma

dagarwal_splunk
Splunk Employee
Splunk Employee

I think this might work:
1. In SH and Indexers you will have collectd running (pointing to "localhost").
2. SH collectd data forwarded to Indexer using inputs.conf (udp input) and outputs.conf (https://answers.splunk.com/answers/4209/search-head-configured-as-a-forwarder.html) OR install a new UF ?
3. Indexer you add inputs.conf (udp input). You should already have the SAI Add-on here.

0 Karma

ntankersley_spl
Splunk Employee
Splunk Employee

You can use the config files for collectd on your monitored hosts, look under /etc/collectd/collectd.conf (Most Linux and Unix) or /etc/collectd.conf (RHEL)

0 Karma

myfriendhenry
Explorer

I understand the collectd part, np. I use the manual method and open a udp port as in "https://docs.splunk.com/Documentation/InfraApp/1.3.0/Admin/WriteCollectdToUF" and it works perfectly for my universal forwarders.
So I got all of that.
My confusion is WHERE to put INPUTS.CONF on the SEARCHHEADS and INDEXERS to monitor THEM with the app.

0 Karma
Get Updates on the Splunk Community!

New Dates, New City: Save the Date for .conf25!

Wake up, babe! New .conf25 dates AND location just dropped!! That's right, this year, .conf25 is taking place ...

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...