topic Re: List all Splunk Servers with a REST search command in Splunk Search

List all Splunk Servers with a REST search command

edoardo_vicendo — Wed, 01 Jul 2020 16:31:44 GMT

I know this has been probably asked before, but I didn't found an answer yet.

Is there any way to know which are all the Splunk Servers (Search Heads, Indexers, Deployment Server, Master Node etc..) in a distributed environment with a REST call made via SPL? It would be grateful if the query does not have to be run from the Server instance where the Monitoring Console is enabled.

Basically I would need to have the same result you can get clicking on "Instances" in the Monitoring Console.

Thanks a lot,
Edoardo

Re: List all Splunk Servers with a REST search command

richgalloway — Wed, 01 Jul 2020 16:46:51 GMT

The Monitoring Console is the one point that is aware of all Splunk instances. If you can't or won't use that then you may be out of luck.

Re: List all Splunk Servers with a REST search command

isoutamo — Wed, 01 Jul 2020 17:08:32 GMT

As @richgalloway said you couldn’t get that information with REST. I propose to you to look this guide https://docs.splunk.com/Documentation/Splunk/8.0.4/InheritedDeployment/Introduction to get more information about your environment.

r. Ismo

Re: List all Splunk Servers with a REST search command

edoardo_vicendo — Thu, 02 Jul 2020 08:10:56 GMT

@richgalloway

Thanks for your feedback.

Do you think that from the Search Head, with the SPL rest command, is it possible to perform a rest call to the Monitoring console to get the values showed on "Instances" view?

Re: List all Splunk Servers with a REST search command

edoardo_vicendo — Thu, 02 Jul 2020 08:14:25 GMT

@isoutamo

Thank you for the very useful documentation, I'll keep in mind. Unfortunately it does not solve my problem

Re: List all Splunk Servers with a REST search command

isoutamo — Thu, 02 Jul 2020 08:53:00 GMT

This should explain what you can query with SPL rest query

https://community.splunk.com/t5/Getting-Data-In/Can-we-use-quot-rest-quot-to-call-Splunk-instances-other-than/td-p/162851

Re: List all Splunk Servers with a REST search command

isoutamo — Thu, 02 Jul 2020 09:28:19 GMT

If all nodes in your environment are sending their internal logs to indexers then you can figure out those information (at least most, maybe not all?) from couple of queries from _internal and _introspection. Probably it's good to create some lookups with scheduled searches to put correct group information to explain node's roles.

Basically MC is stored all those instance roles, groups etc. to lookups when you apply configuration it's configuration page.

Some queries:
# SHC clusters and members
index=_internal sourcetype=splunkd component=CMPeer
| stats values(host) AS CM by peer_name

# Indexer clusters with peers
index=_internal sourcetype=splunkd component=SHCMaster
| stats values(host) as SHC_Captain by peer

# All nodes
index=_internal sourcetype=splunkd component=ExecProcessor command=Python
| stats count by host

Then OS, mem, cpu etc. info can be found from _introspection.
Maybe there is already some apps for this on splunkbase?

r. Ismo

Re: List all Splunk Servers with a REST search command

edoardo_vicendo — Thu, 02 Jul 2020 09:46:21 GMT

@isoutamo

Thank you, I know how to do a SPL REST call, the point is that I don't know which is (or which are) the REST calls to be done to list all the servers within a Splunk deployment

Re: List all Splunk Servers with a REST search command

edoardo_vicendo — Thu, 02 Jul 2020 09:56:10 GMT

In the meanwhile I found this, if executed from the Monitoring Console it reports all the Splunk Servers:

| rest /services/server/status count=0 splunk_server=* | dedup splunk_server | table splunk_server

Now need to check if the same REST call can be done from the Search Head to the Monitoring Console (that in my deployment is in a different server).

Re: List all Splunk Servers with a REST search command

richgalloway — Thu, 02 Jul 2020 12:54:45 GMT

REST calls are processed by the local search head and by all search peers. Since the MC is not a search peer, it will not participate in the query.

Re: List all Splunk Servers with a REST search command

chrisboy68 — Thu, 02 Jul 2020 19:44:41 GMT

This is what the MC uses:

| rest splunk_server_group=* splunk_server_group="*" /services/server/status

you can remove the first two parms if not running on the MC and it will return everything.

HTH

Chris

Re: List all Splunk Servers with a REST search command

isoutamo — Thu, 02 Jul 2020 19:54:58 GMT

In MC you have defined all other nodes as peers to get their information to MC. This is not a situation (or should not) on any other SHs. As it’s said earlier REST get information only on local node (splunk_server=local) or all search peers which it use to query normally. For that reason this is working on MC, but not on other SHs

R. Ismo