Splunk Enterprise Security 5.0 warning messages

[kwokkal]

Hi,
Newbie here. We recently had professional services setup and installed Splunk a few months back. It's been running fine until recently.
On our Splunk Enterprise Security server, we have the following messages:
3/27/2018, 10:29:55 AM
Search peer splunkentsec01 has the following message: Health Check: msg="A script is in an unknown state" input="../bin/collector.path" stanza="default" Learn more.

3/27/2018, 10:29:55 AM
Search peer splunkentsec01 has the following message: Health Check: msg="A script is in an unknown state" input="./opt/splunk/etc/apps/SA-Utils/bin/configuration_check.py" stanza="configuration_check://confcheck_es_migrate_reviewstatus_transitions

It is then followed by messages stating that our three Indexers do not meet minimum system requirements which I do not believe is the case.
Search peer splunkentsec01 has the following message: Health Check: The list of indexes to be searched by default by the admin role on Splunk server "splunkentsec01" includes all non-internal indexes which might cause performance problems. Learn more.

3/27/2018, 10:29:55 AM
Search peer splunkentsec01 has the following message: Health Check: Splunk server "splunkindexer03" does not meet the recommended minimum system requirements. Learn more.

3/27/2018, 10:29:55 AM
Search peer splunkentsec01 has the following message: Health Check: Splunk server "splunkentsec01" does not meet the recommended minimum system requirements. Learn more.

3/27/2018, 10:29:55 AM
Search peer splunkentsec01 has the following message: Health Check: Splunk server "splunkindexer01" does not meet the recommended minimum system requirements. Learn more.

3/27/2018, 10:29:55 AM
Search peer splunkentsec01 has the following message: Health Check: Splunk server "splunkindexer02" does not meet the recommended minimum system requirements. Learn more.

The "learn more" links did lead me to a KB article. But it seems more to do with suppressing the messages than actually determining the root cause and fixing the problem.

Any guidance and help would be appreciated.
Thanks in advance.

[kwokkal]

Hi,
I believe there is a workaround in place and it's suppose to be a "known workaround".
Please reference this link: https://docs.splunk.com/Documentation/ES/5.0.0/RN/KnownIssues

In the link, please reference "Issue Number SOLNESS-14947, SOLNESS-15058"

By making the change, it seems to help resolve the warnings.
Hope this helps.

[dharveynswccd]

Thank you kwokkal. I searched in Splunk web for the "Audit - Script Errors" search string. I then modified and saved it according to the workaround suggested in SOLNESS-14947, SOLNESS-15058. Hopefully I've done that correctly and I didn't break Splunk. We will find out soon!

[kwokkal]

Cool! Let me know if the workaround works for you as it did for me. I would be interested in knowing the workaround is for multiple environments.

[dharveynswccd]

@kwokkal, unfortunately that did not work for me even though I didn't break Splunk so I may not have effected the workaround correctly. Did you do it the same way I did or was there some other way you performed the workaround?

[kwokkal]

Hi,
We replaced the existing string with the workaround string. But I would recommend you copy your existing string to a notepad file and save it as a backup.

This is where we made the change on the Enterprise Security 5 server:
You can go to Settings->Searches, Reports, and Alerts->Do a search for "Audit - Script Errors" -> Edit -> Edit Alert

You'll see an existing string there and you can back that up.
Once it's backed up, replace it with the entire workaround string.

That did the trick for us.

[dharveynswccd]

Was basically what I did.
Did it again this morning and got the error: "Error in 'where' command: The operator at '{noformat}' is invalid."

[kwokkal]

Hi. Sorry. I don't think I will be able to help further. The workaround worked for us and there may be something specific to your environment where the string needs to be further modified.

My suggestion is to double check the string doesn't have extra spacing and is copied exactly as in the Splunk site. If it still doesn't help, open a case with Splunk support. They may be able to see something that's not visible here by having them review logs.

[dharveynswccd]

Thanks dude. At least you got me pointed in the right direction.

[rvany]

Don't copy {noformat} - I believe this is a splunk answers formatting directive. Just start copying with the "| rest..." and end at "...| where isnotnull(errmsg) AND ignore=0".

I compared the old and the new command. Somewhere near the middle of the string you had:

OLD:
... started=if(key=="time closed", value, started), stopped=if(key=="time opened", value, stopped) ...

And after copying you have:

NEW:
... started=if(key=="time opened", value, started), stopped=if(key=="time closed", value, stopped) ...

Just the bold words were changed.

[smoir_splunk]

Those messages might have been suppressed before the upgrade, and are now showing up because the method of displaying the messages changed. If your indexers meet the minimum system requirements listed on the page linked from the message, then those messages can be ignored (and might be getting displayed in error). The three messages should lead to three separate documentation pages about how to mitigate the issues, and only how to suppress if needed.

[kwokkal]

Hi,
I don't think we performed an upgrade. It was a clean install of 6.5 from the ground up. Everything was fine until recently.

The link pages explains how to suppress the messages. But one of the questions I have is doesn't it defeat the purpose by suppressing the messages? I was hoping we can find the root cause of the actual problem and fix it.

Thank you for your input and your response!

[dharveynswccd]

Good morning guys. I too upgraded ESS to 5.0 and suddenly started receiving the same alerts that KWOKKAL is receiving. Was not receiving the alerts prior to the upgrade.I am suppressing the messages for now because they are getting outrageous, and my Indexers also meet/exceed the necessary requirements.

[dharveynswccd]

Hi,
I'm wondering if Splunk has a concrete solution to this. I as well would not only like to suppress the messages. I would like to know if these messages are false positive readings or not. If true, how do I resolve? Here is an example of one of the messages I received today: 'Health Check: msg="A script is in an unknown state" input="../bin/collector.path" stanza="default"'.

Thanks Splunkers

[smoir_splunk]

If you click on the link in the message it should take you to a documentation page that will help you troubleshoot the errors.