Splunk Search

Splunk Health Check (Warning, Info and N/A)

vino06
New Member

Hi Guys,

Good Day!

Regarding on our Splunk servers, we've performed a health check and we found some warning, info and n/a status. Just want to ask what this are and how this result impacts on our application. Please see below screenshot.

Thank you!
Kevin

0 Karma

mattymo
Splunk Employee
Splunk Employee

Hi Kevin!

The actions you should take are dependent on which nodes are involved and require some knowledge about your environment, but I'll do my best to set you in the right direction here.

ULIMITS & THP
For the System and Environment warnings, it is telling you that THP and ulimits are not optimally set for a Splunk Enterprise instance. I would only be worried about correcting this in your core Splunk nodes (Indexers, Search Heads, etc....anything other than Universal Forwarders, really).

Here is some documentation on THP and Ulimits. How you these depends on your system, so work with your sysadmins to ensure they are set persistently.

About Ulimit - https://docs.splunk.com/Documentation/Splunk/6.6.2/Troubleshooting/ulimitErrors
Example How to adjust - https://www.tecmint.com/increase-set-open-file-limits-in-linux/
* note in the versions of RHEL i have played with lately, setting the ulimits on boot-start can be tricky, consult your vendor docs if necessary.

About THP - https://docs.splunk.com/Documentation/Splunk/6.6.2/ReleaseNotes/SplunkandTHP
Example How to Disable THP in Centos 7 - https://newbiedba.wordpress.com/2015/09/07/disabling-transparent-huge-pages-in-centos-7-x/

Missing Forwarders

Missing forwarders is simply forwarders that have not been seen in the last 15 minutes by your indexers. You can see more on this in Forwarder Management dashboard and can be resolved by rebuilding your forwarder lookup if need be. This may or may not be impacting you depending on whether they are decommissioned servers, etc. You will need to

Skipped Searches

This is something you will need to look at in your search heads. You can use the Monitoring Console to analyze your search performance. There is a good break down on what is being skipped under Settings > Monitoring Console > Search > Search Activity: Instance

- MattyMo
Get Updates on the Splunk Community!

Celebrating Fast Lane: 2025 Authorized Learning Partner of the Year

At .conf25, Splunk proudly recognized Fast Lane as the 2025 Authorized Learning Partner of the Year. This ...

Tech Talk Recap | Mastering Threat Hunting

Mastering Threat HuntingDive into the world of threat hunting, exploring the key differences between ...

Observability for AI Applications: Troubleshooting Latency

If you’re working with proprietary company data, you’re probably going to have a locally hosted LLM or many ...