Splunk Enterprise

Cannot Disable Health Report Features in 8.2.2

kisstian
Explorer

I am not sure if anyone else has encountered this, but in our distributed environment that was just upgraded from 8.0.3 to 8.2.2, we have noticed issues with the health report manager.  The new IOWait feature in the health report is extremely "chatty" even though all other aspects of the deployment are in great shape.  Even though we can successfully disable the IOWait feature in the console and via a local health.conf file, the feature is still being included in the health report.  I've opened up a case with Splunk support, but was just wondering if anyone else has encountered this behavior.

Labels (2)

pellegrini
Path Finder

Did you manage to resolve this issue?

You need to change the thresholds, or disable iowait, on each enterprise instance. See Answers https://community.splunk.com/t5/Splunk-Enterprise/Where-do-I-configure-the-health-conf-so-that-I-can...

0 Karma

nunoaragao
Explorer

Just got an answer from Splunk Support, @kisstian  / @samjenk_2  / everyone

Splunk have now updated their documentation regarding disable health report features.
It states in a box:

If distributed health reporting is enabled for your deployment, disabling a feature on the local instance will not be reflected in the health report.

It seems, the workaround to disable a feature in +8.2 has just became a feature. The old behavior in +8.1 in which you could disable a single feature regardless of distributed health report has been "improved"

If anyone submits an Idea to bring the old behavior back, let me know. You get my vote.

 

0 Karma

samjenk_2
Explorer

Thanks to you all for reporting this. We experienced the same issue in our upgrade from 8.0.3 to 8.2.2.1. Our Splunk support contact confirmed they're aware of it, and for now we're hoping for a expeditious resolution rather than disabling reporting on it.

Question for @nunoaragao, is there a public page where we can track the status of SPL-213405? I was hoping it was a public bug report, but not being able to find it with Google, I gather that's the ticket number for the case you opened with support?

nunoaragao
Explorer

Hi @samjenk_2 , my case(s) with Splunk Support were #2733102 and #2737559 .. SPL-213405 is Splunk's internal JIRA to track this issue. It may, or not, then show up on Splunk's release notes as known issue. It's still being investigated. If you deal with Support you can ask to link with it.

My issue is that Docs say "You can disable any feature (...) for example, if you want to exclude a feature's status from the health report". So we expect to be able to disable a specific feature (i.e. Buckets) without requiring to disable distributed_health_reporter, which would also disable/hide a lot of other features if we're on a typical topology where we have search head clusters and clustered indexers. In other words, tell the Search Head to gray out a Indexer peer feature even if that peer is reporting health.

It matches your scenario ?

0 Karma

samjenk_2
Explorer

Thanks for the ticket references, @nunoaragao. I just heard from our Splunk administrator that Splunk has gotten back to him on this matter. With the caveat that your Splunk deployment (and the deployment of others) may be different from ours, they told us that the IOWait thresholds were simply set too aggressively, and that we can either disable the health report like others are doing, or tune the IOWait thresholds in health.conf, as described here:

https://docs.splunk.com/Documentation/Splunk/8.2.3/DMC/Configurefeaturemonitoring

Disappointingly, it looks like the example health.conf file in Splunk's online documentation [1] doesn't include a description of the iowait feature stanza, but it's reasonably documented in the file $SPLUNK/etc/system/default/health.conf. There are several different metrics tracked in this feature, and in our file, they all have to do with CPU utilization (I would have thought disc I/O).

Personally, I'm inclined to monitor CPU utilization with other tools to work out what an typical load is like for these metrics, and adjust the thresholds accordingly. Or maybe disable the IOWait feature, but leave the rest of the health features enabled.

[1] https://docs.splunk.com/Documentation/Splunk/8.2.3/Admin/Healthconf#health.conf.example

0 Karma

nunoaragao
Explorer

Hi @samjenk_2 ,

These are the events used to toggle health colours in Splunk's Health Report.
So we get to know what are the typical values, and in which servers.

index=_introspection sourcetype=splunk_resource_usage component=IOWait 

 

0 Karma

nunoaragao
Explorer

I also found this issue on 8.2.2.1, and Splunk is tracking this on SPL-213405.

My issue is that I disable IOWait and Buckets health check on the Search Heads, but as long as search peers report issues with these features, they'll still show up. Except if we disable distributed health report, but then we lose visibility on a lot of other health checks.

And apparently, the documentation now reports that distributed health report is enabled by default.

0 Karma

brad_thomas
Explorer

I noticed the same thing upon upgrading to 8.2.2 with IOWait. 

I had to disable the IOWait health check on the Search Head and also on the Indexers, but it sounds like disabling the distributed health report would be a better solution. 

0 Karma

kisstian
Explorer

Just in case anyone else experiences this issue, what I did to resolve it was disable the distributed health report which is enabled by default in 8.2.  Once done, I was able to successfully disable the feature  in the health report manager.

splunkreal
Motivator

Hello,

https://docs.splunk.com/Documentation/Splunk/8.2.2/DMC/Aboutfeaturemonitoring : "By default, the distributed health report is disabled." do you speak about same setting?

Thanks.

 

* If this helps, please upvote or accept solution if it solved *
0 Karma

kisstian
Explorer

Yes, that is what I was talking about.  Even though the documentation indicates that the distributed report is disabled by default, I noticed in my environment that after upgrading from 8.0.3 to 8.2.2, the feature was enabled everywhere.

splunkreal
Motivator

Hello,

looks like documentation has been updated : https://docs.splunk.com/Documentation/Splunk/8.2.2/DMC/Aboutfeaturemonitoring

"By default, the distributed health report is enabled (set to disabled = 0) in health.conf"

* If this helps, please upvote or accept solution if it solved *
0 Karma

splunkreal
Motivator

Thanks it works, did you log support case regarding this?

Also this surely removes lots of interesting features...

* If this helps, please upvote or accept solution if it solved *
0 Karma

kisstian
Explorer

Excellent, I am glad I was able to assist.  I did open a ticket and included this information within it.

0 Karma

splunkreal
Motivator

Nice good, please keep us updated!

* If this helps, please upvote or accept solution if it solved *
0 Karma
Get Updates on the Splunk Community!

New Dates, New City: Save the Date for .conf25!

Wake up, babe! New .conf25 dates AND location just dropped!! That's right, this year, .conf25 is taking place ...

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...