Splunk Health Check (Warning, Info and N/A)

vino06 — Fri, 21 Jul 2017 03:51:24 GMT

Hi Guys,

Good Day!

Regarding on our Splunk servers, we've performed a health check and we found some warning, info and n/a status. Just want to ask what this are and how this result impacts on our application. Please see below screenshot.

Thank you!
Kevin

Re: Splunk Health Check (Warning, Info and N/A)

mattymo — Fri, 21 Jul 2017 14:18:23 GMT

Hi Kevin!

The actions you should take are dependent on which nodes are involved and require some knowledge about your environment, but I'll do my best to set you in the right direction here.

ULIMITS & THP
For the System and Environment warnings, it is telling you that THP and ulimits are not optimally set for a Splunk Enterprise instance. I would only be worried about correcting this in your core Splunk nodes (Indexers, Search Heads, etc....anything other than Universal Forwarders, really).

Here is some documentation on THP and Ulimits. How you these depends on your system, so work with your sysadmins to ensure they are set persistently.

About Ulimit - https://docs.splunk.com/Documentation/Splunk/6.6.2/Troubleshooting/ulimitErrors
Example How to adjust - https://www.tecmint.com/increase-set-open-file-limits-in-linux/
* note in the versions of RHEL i have played with lately, setting the ulimits on boot-start can be tricky, consult your vendor docs if necessary.

About THP - https://docs.splunk.com/Documentation/Splunk/6.6.2/ReleaseNotes/SplunkandTHP
Example How to Disable THP in Centos 7 - https://newbiedba.wordpress.com/2015/09/07/disabling-transparent-huge-pages-in-centos-7-x/

Missing Forwarders

Missing forwarders is simply forwarders that have not been seen in the last 15 minutes by your indexers. You can see more on this in Forwarder Management dashboard and can be resolved by rebuilding your forwarder lookup if need be. This may or may not be impacting you depending on whether they are decommissioned servers, etc. You will need to

Skipped Searches

This is something you will need to look at in your search heads. You can use the Monitoring Console to analyze your search performance. There is a good break down on what is being skipped under Settings > Monitoring Console > Search > Search Activity: Instance

topic Splunk Health Check (Warning, Info and N/A) in Splunk Search

Splunk Health Check (Warning, Info and N/A)

Re: Splunk Health Check (Warning, Info and N/A)