Splunk Health Check (Warning, Info and N/A)

[vino06]

Hi Guys,

Good Day!

Regarding on our Splunk servers, we've performed a health check and we found some warning, info and n/a status. Just want to ask what this are and how this result impacts on our application. Please see below screenshot.

Thank you!
Kevin

[mattymo]

Hi Kevin!

The actions you should take are dependent on which nodes are involved and require some knowledge about your environment, but I'll do my best to set you in the right direction here.

ULIMITS & THP
For the System and Environment warnings, it is telling you that THP and ulimits are not optimally set for a Splunk Enterprise instance. I would only be worried about correcting this in your core Splunk nodes (Indexers, Search Heads, etc....anything other than Universal Forwarders, really).

Here is some documentation on THP and Ulimits. How you these depends on your system, so work with your sysadmins to ensure they are set persistently.

About Ulimit - https://docs.splunk.com/Documentation/Splunk/6.6.2/Troubleshooting/ulimitErrors
Example How to adjust - https://www.tecmint.com/increase-set-open-file-limits-in-linux/
* note in the versions of RHEL i have played with lately, setting the ulimits on boot-start can be tricky, consult your vendor docs if necessary.

About THP - https://docs.splunk.com/Documentation/Splunk/6.6.2/ReleaseNotes/SplunkandTHP
Example How to Disable THP in Centos 7 - https://newbiedba.wordpress.com/2015/09/07/disabling-transparent-huge-pages-in-centos-7-x/

Missing Forwarders

Missing forwarders is simply forwarders that have not been seen in the last 15 minutes by your indexers. You can see more on this in Forwarder Management dashboard and can be resolved by rebuilding your forwarder lookup if need be. This may or may not be impacting you depending on whether they are decommissioned servers, etc. You will need to

Skipped Searches

This is something you will need to look at in your search heads. You can use the Monitoring Console to analyze your search performance. There is a good break down on what is being skipped under Settings > Monitoring Console > Search > Search Activity: Instance