Splunk Enterprise Security

"One or more machines does not meet the recommended minimum system requirements. Review the documentation for details." How do I get rid of this message?

brent_weaver
Builder

Good morning. I am constantly getting the message:

One or more machines does not meet the recommended minimum system requirements. Review the documentation for details.

How do I get it to stop, regardless of my system requirements? This is very annoying and causes alarm to our Network Operations Center. Any advice is MUCH appreciated.

jizzmaster
Path Finder

Renaming the file, as my previous answer suggested, does not stop the alerts. What appears to work is editing the search query that triggers this alert.

  1. Go to /opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/default/savedsearches.conf
  2. Copy out the [Audit - ES System Requirements] stanza
  3. Paste it into /opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/local/savedsearches.conf
  4. Edit the appropriate test, such as "numberOfCores" or "physicalMemoryMB"

jizzmaster
Path Finder

A Python script performs the check that kicks out the alert in question.

/opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/bin/configuration_checks/confcheck_es_system_requirements.py

Renaming this file (to something like confcheck_es_system_requirements.bak) seems to effectively stop in from running and stop the alerts.

While it is not the proper way to run Splunk, this addresses the question asked. I'm not going to tell you how to run your Splunk instance. You know your environment better than anyone else here. We all know what the term "minimum requirements" means. The question is not asking for people to tell him to upgrade to the minimum requirements, which is the straightforward alert verbiage.

0 Karma

dhathaway
Engager

I downvoted this post because author later stated the answer does not work

0 Karma

dwaddle
SplunkTrust
SplunkTrust

So, as you've probably already seen, ES is running this search to determine if "minimum requirements" are met:

| rest splunk_server=* count=0 /services/server/info | search ((server_roles="search_head" AND (numberOfCores<16 OR physicalMemoryMB<16000)) OR (server_roles="indexer" AND (numberOfCores<12 OR physicalMemoryMB<12000))) | table splunk_server,server_roles,numberOfCores,physicalMemoryMB

It expects search heads to have 16 cores / 16000MB of RAM, and indexers to have 12 cores / 12000GB of RAM. These are really not "suggestions" or "it'll run best with" or "optimal specifications". Trying to run ES with fewer resources than these will cause things not to work. Trying to fix it with "racing tape" is just not advised.

That being said, if you still want to "disable" the message then modify the search Audit - ES System Requirements above (under the SplunkEnterpriseSecuritySuite app) to lower the minimum limits. If you disable this search, then you'll get even more system messages about configuration_check://confcheck_es_system_requirements exiting with code 3.

jrodman
Splunk Employee
Splunk Employee

Indeed, I've worked through multi-week-long painful system diagnosis problems where customers were enduring outages and brownouts due to running over capacity. Determining the full set of necessary work, the physical configurations, etc of the entire search environment, as well as the workload of user configuration as well as the aggregate of all installed apps is just not quick work without knowing the environment.

Messages like this will save tens of thousands of dollars in waste.

jizzmaster
Path Finder

I downvoted this post because it doesn't answer the question that was asked.

0 Karma

lguinn2
Legend

@jizzmaster - while I appreciate that you gave a reason for your downvote, I disagree. In this forum, the custom is to (1) only downvote for wrong answers that will cause problems if implemented and (2) to give a cogent reason for your downvote. Finally, you should provide a better answer (if you can) to replace the down-voted answer.

@dwaddle's answer went beyond the actual question to the underlying problem. While @dwaddle's answer technically does not answer the question, it provides the proper solution because going below minimum resources in Splunk can cause many problems, some of which may not be detected immediately.

jizzmaster
Path Finder

The answer went "beyond" the question by not answering the actual question. Read the question, "How do I get it to stop, regardless of my system requirements?" I repeat, "regardless of my system requirements." This site is called answers.splunk.com, not Im-gonna-tell-you-how-to-run-your-splunk-instance-because-I-know-your-environment-and-needs-better-than-you.splunk.com. It did not answer the question therefore it should not be considered an answer.

I'm not disagreeing on the points made about the effectiveness of the documented minimum requirements. They are sound. And belong in the answers for a question asking if it is ok to run a Splunk box under the minimum requirements. But that is not this question here.

0 Karma

dwaddle
SplunkTrust
SplunkTrust

A fair critique. But, sometimes you have to tell someone that what they're asking to do is a bad idea. Like http://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags. There is a good reason to not tell someone how to subtly break a very expensive product in ways that are not upgrade safe. I appreciate the feedback and the candor, but must disagree with your assessment.

brent_weaver
Builder

dwaddle -

Thank you for the response. I love the term "racing tape", such a great visual! 🙂

All I did was disable the check functionally in Splunk and then reenabled it.

0 Karma

Richfez
SplunkTrust
SplunkTrust

I'm not surprised your Network Operations Center is alarmed that their systems are below specification.

Splunk's minimum hardware requirements is pretty low, and there's a reason they exist. Below these numbers not only can performance suffer (obviously), but sometimes certain things just won't work. The Deployment Server is a great example: with a 2 core, 4 GB box, there are times you simply can't reload the deploy-server portion. It will simply fail and nothing in the error logs indicate specifically why, and it doesn't even appear to take long. Just total failure. Bump it up to specs, and it works fine.

If it is ES, there's a check that runs (or used to, it's probably still there though) called "Audit - ES System Requirements" in the Configuration Section of the Data Inputs that you coudl disable or modify. Otherwise, I'm not aware of anything. Support may be able to help.

brent_weaver
Builder

The error I am getting is as follows:

msg="A script exited abnormally" input="/opt/splunk/etc/apps/SA-Utils/bin/configuration_check.py" stanza="configuration_check://confcheck_es_system_requirements" status="exited with code 3"

I am even more confused as I reenabled the check I had diabled in Alerts and reports. Iis called "Audit - ES System Requirements" .

Any help is much appreciated!

0 Karma

Richfez
SplunkTrust
SplunkTrust

Whatever app SA-Utils is, redownload it (or grab your existing copy of it), unzip/untar it, take a copy of your configuration_check.py and copy it back into place, then restart splunk. You probably oopsed a typo into it, but that should fix it.

brent_weaver
Builder

Rich - I never modified the script at all - just disabled it in Splunk apps?!?!?

0 Karma

brent_weaver
Builder

I disabled this query/report and it gave me a wole new error. It looks like there is a script called /opt/splunk/etc/apps/SA-Utils/bin/configuration_check.py. can I just put an exit 0 at the begining?

0 Karma

Richfez
SplunkTrust
SplunkTrust

Is it telling you in error? What spec are you not meeting?

0 Karma

bosburn_splunk
Splunk Employee
Splunk Employee

Brent,
As a Splunk support engineer, I would highly discourage you from messing with the scripts like that.

If your boxes are under spec'd, and you have an issue, the first thing we will do is tell you to get the boxes up to the minimum specs. Especially if you're running an application that is heavily dependent on box specifications such as Enterprise Security.

Brian

robert_miller
Path Finder

I have the same issue but it appears Splunk is not taking into consideration our hyper threaded cores. I thought running 6.3.x would fix this issue.

0 Karma

jrodman
Splunk Employee
Splunk Employee

Incidentally, hyperthreaded cores will not offer a significant increase in real performance for splunk tasks commensurate to their quantity. The benefit is real, but not linear with hyperthread counts.

0 Karma
Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...