Monitoring Splunk

NOT a question: There is a small bug in the health checks for 6.5.0 Monitoring Console and this is how to resolve it.

lycollicott
Motivator

There appears to be a bug in splunk_monitoring_console\default\checklist.conf.

After running the Health Checks, the GUI drill-down for "Search scheduler skip ratio" states "This checks whether scheduled searches were skipped in the past hour.", but that is not correct. It actually searches all time instead of the last 60 minutes.

I added earliest=-60m to the configuration to resolve it.

1 Solution

lycollicott
Motivator

Just sharing. 🙂

View solution in original post

santiagn
Path Finder

i need to do this but dont know where to add the line inside checklist.conf, can someone show me an example? do i need to restart the search head after?

0 Karma

lycollicott
Motivator

Just sharing. 🙂

View solution in original post

j4adam
Communicator

Thanks for posting this! I was scratching my head at how I had skipped 71879 searches in an hour!

0 Karma

hexx
Splunk Employee
Splunk Employee

Thank you for reporting this issue with the Monitoring Console Health-check!

Product defect SPL-130183 was filed to fix it in a future release.

saurabh009
Path Finder

Is this fixed in 6.5.1?

lycollicott
Motivator

According to http://docs.splunk.com/Documentation/Splunk/latest/ReleaseNotes/Knownissues they are still referring to my work around.

2016-10-12  SPL-130183  

Drilldown search for the "Search scheduler skip ratio" Monitoring Console health-check runs against all time instead of last 60 minutes

Workaround:
You can edit this particular check and add "earliest=-60m" as a search term.
0 Karma

saurabh009
Path Finder

How much time it usually take to complete?
For me it took almost more then hour to complete just "search scheduler skip ratio" after implementing your solution.

0 Karma

j4adam
Communicator

@saurabh009 What's the size of your environment and what are the specs of the machine running the MC. Is it in distributed mode and is it also on a search head?

The MC just searches the splunk internal logs IIRC, so it taking a long time seems weird unless the machine is either underspec or taxed heavily (like running the MC in distributed mode on an active search head)

Just a couple of thoughts!

0 Karma

saurabh009
Path Finder

Yes, MC is in distributed mode but it is deployed only on dedicated machine not used as search head.
I am trying to monitor more than 100 instances, which comprises of various indexers, search heads,ICMs and DSs.

0 Karma

lycollicott
Motivator

Did you have a lot of skipped searches returned?

0 Karma

lycollicott
Motivator

Less than a second is all it took for me.

0 Karma