Getting all forwarder names via a REST call

[ddrillic]

We created a monitoring dashboard (outside of Splunk) which relies on rest /services/deployment/server/clients to get the list of the forwarders in the system. I would expect this call to return the same list as the list within the serverclass.conf file but apparently these two are not compatible. Is there a way to make them compatible? meaning that this REST call would return exactly the list which serverclass.conf holds.

[wrangler2x]

Well, reading this allowed me to be to develop a REST search that can lookup the serverClasses associated with a particular host, which is handy-dandy when you get a decommissioned server notice. Here is the base search:

| rest /services/deployment/server/serverclasses splunk_server=local
| table title whitelist.*
| untable title whitelist hostname
| stats values(title) AS serverClasses count by hostname

The base search gives you a list of all hosts and the serverClasses associated with them. To look for a particular host, add this final line:

| search hostname=FQDNofSomeHost

I wanted to create a dashboard out of this, and so the search looks like this:

 <searchTemplate>
| rest /services/deployment/server/serverclasses splunk_server=local
| table title whitelist.*
| untable title whitelist hostname
| stats values(title) AS serverClasses count by hostname
| search hostname = $HOSTNAME$
</searchTemplate>

With a fieldset clause next:

<input type="text" token="HOSTNAME" searchWhenChanged="true"></input>
<input type="time" searchWhenChanged="true">
  <default>Today</default>
</input>

You can use * as a wildcard to search for all hosts, or as the end of a partial host name.

[somesoni2 · Accepted solution]

You should be able to get list of whitelist servers using following query.

| rest /services/deployment/server/serverclasses | table title whitelist.* | untable title whitelist hostname | stats count by hostname | table hostname

You can put it to in a lookup file or just use the rest query itself and compare it against deployment/server/clients to know which clients that are configured in serverclass.conf but not sending phonehome. Something like this

| rest /services/deployment/server/serverclasses | table title whitelist.* | untable title whitelist hostname | stats count by hostname | table hostname | eval state="configured" | append [| rest /services/deployment/server/clients | table title | rename title as hostname | eval state="phonehome" ] | stats values(state) as state by hostname | where mvcount(state)=1 AND state="configured"

[ddrillic]

You are truly amazing - can you please convert it to an answer?

[somesoni2]

The REST Api endpoint will give list of client which are phoning home to deployment server. The serverclass.conf file can have many clients which are not connecting to deployment server and/or have wildcarded server names so there many not be one-to-one mapping available. What is you concern on using just the output of REST endpoint?

[ddrillic]

The thing is that we reach situations where forwarders are down for some time and the rest call doesn't list these servers, so they can end up being down for a couple of weeks. We also make a good effort to maintain our serverclass.conf file. So, it is the single source of truth for us about the host inventory. It's interesting as we don't use wildcards when we specify the whitelists. Should I create a lookup table based on the serverclass.conf file and check each entry by itself to see whether it's phoning home?