Monitoring Splunk

What is the process "splunkd instrument-resource-usage -p 8089 --with-kvstore" doing?

leo_wang
Path Finder

Hi ,

Our splunk servers has high CPU usage problem after upgrading to Splunk v6.5

It could related to my previous question :
https://answers.splunk.com/answers/475114/is-there-any-way-to-downgrade-from-65-to-63.html

And I noticed the abnormal high cpu-usage process is
"/opt/splunk/bin/splunkd instrument-resource-usage -p 8089 --with-kvstore"

It is usually use more than 90% CPU all of the time...
What is this process doing for ?
How could I deal with this situation ?

alt text

0 Karma

arowsell_splunk
Splunk Employee
Splunk Employee

Hi,

There is a bug introduced in 6.5:

SPL-133720 - splunkd instrument-resource-usage process uses one full CPU core after upgrade to 6.5.1 on Centos 5

To workaround the issue is 6.5.0, 6.5.1 and 6.5.2 you can disable introspection as per the following:

https://docs.splunk.com/Documentation/Splunk/6.5.2/Troubleshooting/ConfigurePIF

To disable: in $SPLUNK_HOME/etc/apps/introspection_generator_addon/local/app.conf, set:

[install]
state = disabled

This should be fixed in 6.5.3.

0 Karma

inventsekar
Ultra Champion

this process instrument-resource-usage configures the logging on your splunk instance

If you've disabled this logging on your instance, you can still invoke the CLI command. To invoke, at the command line:

$ splunkd instrument-resource-usage [--debug] [--once] [--extra] 

where the flags mean:
--debug: Set logging level to DEBUG (this can also be done via log-cmdline.cfg)
--once: Emit one set of introspection data, and then quit
--extra: This has the same effect as setting acquireExtra_i_data to true in the server.conf [introspection:generator:resource_usage] stanza. See "What gets logged" for which fields are not logged by default and require this flag.

http://docs.splunk.com/Documentation/Splunk/6.5.0/Troubleshooting/ConfigurePIF

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...