Splunk Search

List all Splunk Servers with a REST search command

Communicator

I know this has been probably asked before, but I didn't found an answer yet.

Is there any way to know which are all the Splunk Servers (Search Heads, Indexers, Deployment Server, Master Node etc..) in a distributed environment with a REST call made via SPL? It would be grateful if the query does not have to be run from the Server instance where the Monitoring Console is enabled.

Basically I would need to have the same result you can get clicking on "Instances" in the Monitoring Console.

Thanks a lot,
Edoardo

 

Labels (1)
0 Karma

Contributor

This is what the MC uses:

| rest splunk_server_group=* splunk_server_group="*" /services/server/status

 you can remove the first two parms if not running on the MC and it will return everything. 

HTH

Chris

Champion
In MC you have defined all other nodes as peers to get their information to MC. This is not a situation (or should not) on any other SHs. As it’s said earlier REST get information only on local node (splunk_server=local) or all search peers which it use to query normally. For that reason this is working on MC, but not on other SHs

R. Ismo
0 Karma

SplunkTrust
SplunkTrust
The Monitoring Console is the one point that is aware of all Splunk instances. If you can't or won't use that then you may be out of luck.
---
If this reply helps you, an upvote would be appreciated.

Communicator

@richgalloway 

Thanks for your feedback.

Do you think that from the Search Head, with the SPL rest command, is it possible to perform a rest call to the Monitoring console to get the values showed on "Instances" view?

0 Karma

SplunkTrust
SplunkTrust
REST calls are processed by the local search head and by all search peers. Since the MC is not a search peer, it will not participate in the query.
---
If this reply helps you, an upvote would be appreciated.

Champion
0 Karma

Communicator

@soutamo 

Thank you, I know how to do a SPL REST call, the point is that I don't know which is (or which are) the REST calls to be done to list all the servers within a Splunk deployment

0 Karma

Communicator

In the meanwhile I found this, if executed from the Monitoring Console it reports all the Splunk Servers:

| rest /services/server/status count=0 splunk_server=* | dedup splunk_server | table splunk_server

Now need to check if the same REST call can be done from the Search Head to the Monitoring Console (that in my deployment is in a different server).

0 Karma

Champion
If all nodes in your environment are sending their internal logs to indexers then you can figure out those information (at least most, maybe not all?) from couple of queries from _internal and _introspection. Probably it's good to create some lookups with scheduled searches to put correct group information to explain node's roles.

Basically MC is stored all those instance roles, groups etc. to lookups when you apply configuration it's configuration page.

Some queries:
# SHC clusters and members
index=_internal sourcetype=splunkd component=CMPeer
| stats values(host) AS CM by peer_name

# Indexer clusters with peers
index=_internal sourcetype=splunkd component=SHCMaster
| stats values(host) as SHC_Captain by peer

# All nodes
index=_internal sourcetype=splunkd component=ExecProcessor command=Python
| stats count by host

Then OS, mem, cpu etc. info can be found from _introspection.
Maybe there is already some apps for this on splunkbase?

r. Ismo

Champion

As @richgalloway said you couldn’t get that information with REST. I propose to you to look this guide https://docs.splunk.com/Documentation/Splunk/8.0.4/InheritedDeployment/Introduction to get more information about your environment. 

r. Ismo

Communicator

@soutamo 

Thank you for the very useful documentation, I'll keep in mind. Unfortunately it does not solve my problem

0 Karma