topic Re: Health Check Doesn't Run in Monitoring Splunk

Why isn't Health Check Running?

mello920 — Fri, 22 Apr 2022 16:08:52 GMT

Good Afternoon,

My Splunk Monitoring Console just doesn't seem to work. The Overview or any tab just can't populate their dashboards. I decided to run the Health Check, to see what could be wrong but everything just fails with: "search job stopped unexpectedly". I can search through my index.

I looked into splunkd.log and found no errors that correlate with the Monitoring Console. What could be causing this? Can I reinstall the Monitoring Console?

Any help is greatly appreciated. Thank you.

Re: Health Check Doesn't Run

Stefanie — Thu, 21 Apr 2022 19:41:14 GMT

@mello920
Just to confirm, you are able to run regular searches outside of the Monitoring Console?
Do the other dashboards inside of the Monitoring Console work?

There isn't a way to just 'reinstall' the Monitoring Console by itself. You would have to reinstall Splunk Enterprise.

Re: Health Check Doesn't Run

mello920 — Thu, 21 Apr 2022 19:49:13 GMT

Yes, I'm able to run regular searches outside the Monitoring Console. All the dashboards but the ones under Summary are not working. "Waiting for data" or "Couldn't create search".

I updated the SH from 8.04 to 8.1.9.

Re: Health Check Doesn't Run

Stefanie — Thu, 21 Apr 2022 19:56:14 GMT

Did it work before upgrading?
Could you confirm that the user you are logged into the SH has the "admin_all_objects" capability.

In the Health Check page, if you click on any of the failed items, can you click on the magnifying glass to "open the search" in a new tab? Does that give an error as well?

Re: Health Check Doesn't Run

isoutamo — Thu, 21 Apr 2022 21:13:09 GMT

Can you check that all splunk files on node has owned by your splunk user? If not then you must change those and restart splunk.

r. Ismo

Re: Health Check Doesn't Run

gcusello — Fri, 22 Apr 2022 06:44:30 GMT

Hi @mello920,

a very quick answers to better understand your situation:

are you forwarding logs from all the Splunk servers to Indexers?
where is your Monitor Console: on the Search Head (not a good practice), in a dedicated server or in a server shared with other roles?
if not on the Search Head, did you configured your Monitor Console as a Search Head for your Indexers?
Ciao.

Giuseppe

Re: Health Check Doesn't Run

mello920 — Fri, 22 Apr 2022 15:26:02 GMT

Yes, the Splunk user owns all its files.

Re: Health Check Doesn't Run

mello920 — Fri, 22 Apr 2022 15:33:10 GMT

I'm not a Splunk Admin per se. Our office hasn't had one in a while. The Monitoring Console didn't even load when clicking on it when the SH was on 8.0.4. I performed the upgrade, and soon after I could actually go into the Monitoring Console. Compared to our Prod Env, it just isn't working properly.

Yes, my user role is set with the "admin" rights.

I tried to run the one check for "index status" in the Search App and I get "error in 'rest' command: Invalid argument: '/services/server/introspection/indexer'

Re: Why isn't Health Check Running?

mello920 — Fri, 06 May 2022 06:31:03 GMT

Appears to be related to our F5 WAF flagging the Splunk REST API calls. Our network engineer said they were flagged under "antivirus check". Once an exception to the policy was made, the health checks now run as intended.

Re: Why isn't Health Check Running?

gcusello — Fri, 06 May 2022 06:38:35 GMT

Hi @mello920,

good for you, please accept one answer for the other people of Community (also your one!)

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are appreciated by all the Contributors 😉