Monitoring Splunk

Splunk Rest Commands Invalid

andrew_burnett
Path Finder

I have a HF that was recently expanded in terms of its RAM capacity. Ever since, there has been an issue with REST commands off specific endpoint.

Example:

| rest splunk_server=splunk-hf1 /services/server/info (Works great and gives me the expected results)

| rest splunk_server_group=dmc_group_* /services/server/status/resource-usage/hostwide (When I run it it acts like some of my servers, including a HF has 0 memory)

Labels (1)
0 Karma

andrew_burnett
Path Finder

It makes no difference if I change it and run it on the HF locally itself, the second REST command never works.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

What I have realised with REST api, it could give you a wrong answers if there has happened something "weird" after splunkd has restarted and even sometime if host hasn't rebooted e.g. after volumes/LUNs have added.

So have you try to restart splunkd on those host which you have wrong information? Also do same if you have added mem to host as live operation.

I also suggest you to "disable boot-start" and reenable it after memory changes.

In my test env these seems to work correctly (removed some unneeded lines from output)

[splunk@splunk-demo-rh8] ~>
(0) $ splunk search '| rest splunk_server_group=* /services/server/status/resource-usage/hostwide' -output json|jq|egrep mem
    "mem": "15725.703",
[splunk@splunk-demo-rh8] ~>
(0) $ splunk search '| rest splunk_server=local /services/server/info' -output json|jq|egrep -i mem
    "physicalMemoryMB": "15725",
[splunk@splunk-demo-rh8] ~>
(0) $

Splunk 9.0.5 

 r. Ismo

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Step into “Hunt the Insider: An Splunk ES Premier Mystery” to catch a cybercriminal ...

After a whole week of being on call, you fell asleep on your keyboard, and you hit a sequence of buttons that ...

SplunkTrust Application Period is Officially OPEN!

It's that time, folks! The application/nomination period for the 2026-2027 SplunkTrust is officially open. If ...

Announcing Modern Navigation: A New Era of Splunk User Experience

We are excited to introduce the Modern Navigation feature in the Splunk Platform, available to both cloud and ...