Deployment Architecture

Unknown Memory & CPU Status Search Head Detected on Monitoring Console ( Disk stats are visible)

sdubey_splunk
Splunk Employee
Splunk Employee

Monitoring console : On the monitoring console, two search heads were not reporting CPU/Memory statistics.

It's showing the status as grey. The indexers were reporting all the stats correctly.

We tried below to fix, but with no luck.

  1. We logged into the monitoring console and found two search heads were not reporting Memory and Cpu usage.
  2. We removed the two search heads and added again(Settings ->Distributed Search -> Search Peers : Remove and added)
  3. We still noticed the same issue/error.
  4. When we checked the search head instance details via the monitoring console, we found that only disk status was showing.
  5. We logged into one of the search heads that is not showing cpu/memory stats i. Checked configuration file for "introspection_generator_addon" and added app.conf under "introspection_generator_addon/local" and enabled introspection app.conf : appended [install] state = enabled ii. Restarted Splunk service iii. Still, we do not see resource.log under $SPLUNK_HOME/var/log/introspection iv. We checked and found the "introspection_generator_addon" app was enabled and configured but only collecting disk stats. v. As per "introspection_generator_addon/default/app.conf" : We see that introspection is enabled vi. Upon further checking we found that instrument-resource-usage process was not running.
0 Karma
1 Solution

sdubey_splunk
Splunk Employee
Splunk Employee
  1. Checked documentation and found that introspection is enabled but we were able to see only disk usage logs under $SPLUNK_HOME/var/log/introspection

Upon further checking we found that "script://./bin/collector.path" was disabled. We tried enabling it and that fixed the issue. Hope this helps.
$SPLUNKLE_HOM/etc/apps/introspection_generator_addon/local/inputs.conf
[script://./bin/collector.path]
disabled = 0

View solution in original post

0 Karma

sdubey_splunk
Splunk Employee
Splunk Employee
  1. Checked documentation and found that introspection is enabled but we were able to see only disk usage logs under $SPLUNK_HOME/var/log/introspection

Upon further checking we found that "script://./bin/collector.path" was disabled. We tried enabling it and that fixed the issue. Hope this helps.
$SPLUNKLE_HOM/etc/apps/introspection_generator_addon/local/inputs.conf
[script://./bin/collector.path]
disabled = 0

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...