Splunk Search

List all Splunk Servers with a REST search command

edoardo_vicendo
Contributor

I know this has been probably asked before, but I didn't found an answer yet.

Is there any way to know which are all the Splunk Servers (Search Heads, Indexers, Deployment Server, Master Node etc..) in a distributed environment with a REST call made via SPL? It would be grateful if the query does not have to be run from the Server instance where the Monitoring Console is enabled.

Basically I would need to have the same result you can get clicking on "Instances" in the Monitoring Console.

Thanks a lot,
Edoardo

 

Labels (1)
0 Karma

chrisboy68
Contributor

This is what the MC uses:

| rest splunk_server_group=* splunk_server_group="*" /services/server/status

 you can remove the first two parms if not running on the MC and it will return everything. 

HTH

Chris

isoutamo
SplunkTrust
SplunkTrust
In MC you have defined all other nodes as peers to get their information to MC. This is not a situation (or should not) on any other SHs. As it’s said earlier REST get information only on local node (splunk_server=local) or all search peers which it use to query normally. For that reason this is working on MC, but not on other SHs

R. Ismo
0 Karma

richgalloway
SplunkTrust
SplunkTrust
The Monitoring Console is the one point that is aware of all Splunk instances. If you can't or won't use that then you may be out of luck.
---
If this reply helps you, Karma would be appreciated.

edoardo_vicendo
Contributor

@richgalloway 

Thanks for your feedback.

Do you think that from the Search Head, with the SPL rest command, is it possible to perform a rest call to the Monitoring console to get the values showed on "Instances" view?

0 Karma

richgalloway
SplunkTrust
SplunkTrust
REST calls are processed by the local search head and by all search peers. Since the MC is not a search peer, it will not participate in the query.
---
If this reply helps you, Karma would be appreciated.

isoutamo
SplunkTrust
SplunkTrust
0 Karma

edoardo_vicendo
Contributor

@isoutamo 

Thank you, I know how to do a SPL REST call, the point is that I don't know which is (or which are) the REST calls to be done to list all the servers within a Splunk deployment

0 Karma

edoardo_vicendo
Contributor

In the meanwhile I found this, if executed from the Monitoring Console it reports all the Splunk Servers:

| rest /services/server/status count=0 splunk_server=* | dedup splunk_server | table splunk_server

Now need to check if the same REST call can be done from the Search Head to the Monitoring Console (that in my deployment is in a different server).

0 Karma

isoutamo
SplunkTrust
SplunkTrust
If all nodes in your environment are sending their internal logs to indexers then you can figure out those information (at least most, maybe not all?) from couple of queries from _internal and _introspection. Probably it's good to create some lookups with scheduled searches to put correct group information to explain node's roles.

Basically MC is stored all those instance roles, groups etc. to lookups when you apply configuration it's configuration page.

Some queries:
# SHC clusters and members
index=_internal sourcetype=splunkd component=CMPeer
| stats values(host) AS CM by peer_name

# Indexer clusters with peers
index=_internal sourcetype=splunkd component=SHCMaster
| stats values(host) as SHC_Captain by peer

# All nodes
index=_internal sourcetype=splunkd component=ExecProcessor command=Python
| stats count by host

Then OS, mem, cpu etc. info can be found from _introspection.
Maybe there is already some apps for this on splunkbase?

r. Ismo

isoutamo
SplunkTrust
SplunkTrust

As @richgalloway said you couldn’t get that information with REST. I propose to you to look this guide https://docs.splunk.com/Documentation/Splunk/8.0.4/InheritedDeployment/Introduction to get more information about your environment. 

r. Ismo

edoardo_vicendo
Contributor

@isoutamo 

Thank you for the very useful documentation, I'll keep in mind. Unfortunately it does not solve my problem

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...