Our splunk servers has high CPU usage problem after upgrading to Splunk v6.5
It could related to my previous question :
And I noticed the abnormal high cpu-usage process is
"/opt/splunk/bin/splunkd instrument-resource-usage -p 8089 --with-kvstore"
It is usually use more than 90% CPU all of the time...
What is this process doing for ?
How could I deal with this situation ?
this process instrument-resource-usage configures the logging on your splunk instance
If you've disabled this logging on your instance, you can still invoke the CLI command. To invoke, at the command line:
$ splunkd instrument-resource-usage [--debug] [--once] [--extra]
where the flags mean:
--debug: Set logging level to DEBUG (this can also be done via log-cmdline.cfg)
--once: Emit one set of introspection data, and then quit
--extra: This has the same effect as setting acquireExtraidata to true in the server.conf [introspection:generator:resource_usage] stanza. See "What gets logged" for which fields are not logged by default and require this flag.
There is a bug introduced in 6.5:
SPL-133720 - splunkd instrument-resource-usage process uses one full CPU core after upgrade to 6.5.1 on Centos 5
To workaround the issue is 6.5.0, 6.5.1 and 6.5.2 you can disable introspection as per the following:
To disable: in $SPLUNKHOME/etc/apps/introspectiongenerator_addon/local/app.conf, set:
state = disabled
This should be fixed in 6.5.3.