Why isn't Health Check Running?

[mello920]

Good Afternoon,

My Splunk Monitoring Console just doesn't seem to work. The Overview or any tab just can't populate their dashboards. I decided to run the Health Check, to see what could be wrong but everything just fails with: "search job stopped unexpectedly". I can search through my index.

I looked into splunkd.log and found no errors that correlate with the Monitoring Console. What could be causing this? Can I reinstall the Monitoring Console?

Any help is greatly appreciated. Thank you.

[mello920 · Accepted solution]

Appears to be related to our F5 WAF flagging the Splunk REST API calls. Our network engineer said they were flagged under "antivirus check". Once an exception to the policy was made, the health checks now run as intended.

[gcusello]

Hi @mello920,

good for you, please accept one answer for the other people of Community (also your one!)

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are appreciated by all the Contributors 😉

[gcusello]

Hi @mello920,

a very quick answers to better understand your situation:

• are you forwarding logs from all the Splunk servers to Indexers?
• where is your Monitor Console: on the Search Head (not a good practice), in a dedicated server or in a server shared with other roles?
• if not on the Search Head, did you configured your Monitor Console as a Search Head for your Indexers?
• Ciao.

Giuseppe

[Stefanie]

@mello920 
Just to confirm, you are able to run regular searches outside of the Monitoring Console? 
Do the other dashboards inside of the Monitoring Console work?

There isn't a way to just 'reinstall' the Monitoring Console by itself. You would have to reinstall Splunk Enterprise.

[mello920]

Yes, I'm able to run regular searches outside the Monitoring Console. All the dashboards but the ones under Summary are not working. "Waiting for data" or "Couldn't create search".

I updated the SH from 8.04 to 8.1.9.

[isoutamo]

Can you check that all splunk files on node has owned by your splunk user? If not then you must change those and restart splunk.

r. Ismo

[mello920]

Yes, the Splunk user owns all its files.

[Stefanie]

Did it work before upgrading? 
Could you confirm that the user you are logged into the SH has the "admin_all_objects" capability.

In the Health Check page, if you click on any of the failed items, can you click on the magnifying glass to "open the search" in a new tab? Does that give an error as well?

[mello920]

I'm not a Splunk Admin per se. Our office hasn't had one in a while. The Monitoring Console didn't even load when clicking on it when the SH was on 8.0.4. I performed the upgrade, and soon after I could actually go into the Monitoring Console. Compared to our Prod Env, it just isn't working properly.

Yes, my user role is set with the "admin" rights.

I tried to run the one check for "index status" in the Search App and I get "error in 'rest' command: Invalid argument: '/services/server/introspection/indexer'