Monitoring Splunk

Why isn't Health Check Running?

mello920
Path Finder

Good Afternoon,

My Splunk Monitoring Console just doesn't seem to work. The Overview or any tab just can't populate their dashboards. I decided to run the Health Check, to see what could be wrong but everything just fails with: "search job stopped unexpectedly". I can search through my index.

I looked into splunkd.log and found no errors that correlate with the Monitoring Console. What could be causing this? Can I reinstall the Monitoring Console?

Any help is greatly appreciated. Thank you.

 

Monitor Console Health Check Error.png

Labels (1)
0 Karma
1 Solution

mello920
Path Finder

Appears to be related to our F5 WAF flagging the Splunk REST API calls. Our network engineer said they were flagged under "antivirus check". Once an exception to the policy was made, the health checks now run as intended.

View solution in original post

0 Karma

mello920
Path Finder

Appears to be related to our F5 WAF flagging the Splunk REST API calls. Our network engineer said they were flagged under "antivirus check". Once an exception to the policy was made, the health checks now run as intended.

0 Karma

gcusello
Legend

Hi @mello920,

good for you, please accept one answer for the other people of Community (also your one!)

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are appreciated by all the Contributors 😉

gcusello
Legend

Hi @mello920,

a very quick answers to better understand your situation:

  • are you forwarding logs from all the Splunk servers to Indexers?
  • where is your Monitor Console: on the Search Head (not a good practice), in a dedicated server or in a server shared with other roles?
  • if not on the Search Head, did you configured your Monitor Console as a Search Head for your Indexers?
  • Ciao.

Giuseppe

0 Karma

Stefanie
Builder

@mello920 
Just to confirm, you are able to run regular searches outside of the Monitoring Console? 
Do the other dashboards inside of the Monitoring Console work?

 

There isn't a way to just 'reinstall' the Monitoring Console by itself. You would have to reinstall Splunk Enterprise.

0 Karma

mello920
Path Finder

Yes, I'm able to run regular searches outside the Monitoring Console. All the dashboards but the ones under Summary are not working. "Waiting for data" or "Couldn't create search".

I updated the SH from 8.04 to 8.1.9.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Can you check that all splunk files on node has owned by your splunk user? If not then you must change those and restart splunk.

r. Ismo

0 Karma

mello920
Path Finder

Yes, the Splunk user owns all its files.

0 Karma

Stefanie
Builder

Did it work before upgrading? 
Could you confirm that the user you are logged into the SH has the "admin_all_objects" capability.

 

In the Health Check page, if you click on any of the failed items, can you click on the magnifying glass to "open the search" in a new tab? Does that give an error as well?

0 Karma

mello920
Path Finder

I'm not a Splunk Admin per se. Our office hasn't had one in a while. The Monitoring Console didn't even load when clicking on it when the SH was on 8.0.4. I performed the upgrade, and soon after I could actually go into the Monitoring Console. Compared to our Prod Env, it just isn't working properly.

Yes, my user role is set with the "admin" rights.

I tried to run the one check for "index status" in the Search App and I get "error in 'rest' command: Invalid argument: '/services/server/introspection/indexer'

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...