What is happening in Splunk Enterprise V9.1.0.1 ?

[apietersen]

RE: Case #3270697 After upgrade to 9.1.01 not able to send emails eg. of critical alerts! [ ref:_00D409oyL._5005a2bGRKI:ref ]

After upgrade to v9.1.0.1 Splunk Enterprise, (single instance), last weekend (15 Juli 2023) + changing admin password as was suggested by Assist (which throws an error now !?)

1) Message when using sendemail:

Smpt setting: O365

Checked the login on O365, ofcourse

2) Assist stopped running???

3) Also:

3a

3b

4) new GUI / layout ?

5) Annoying and not working “Don’t show this again” message on every page. Just stepping to another dashboard on the same server/domain ?? 

6) endless waiting:

What is next?

Anyone else suffering from the same issues?

[psecure]

Hello all,

I have a problem with my configuration smtp.
When I send e-mail I get this error :

2024-02-14 16:44:15,213 +0100 ERROR cli_common:482 - Failed to decrypt value: ***************************=, error: Read custom key data size=30

Someone has an idea?

[isoutamo]

Quite probably your key for encrypt has changed? You could try to update old email password via gui and encrypt it again.

[psecure]

Thank @isoutamo ,
Yes, I update email password via gui, and reboot splunk, but this problem is still present

[isoutamo]

You should replace mycert.pem with your own cert file. That name you could see by previous btool. Use also full path to this file. Sometimes it’s also needed to add “splunk cmd OpenSSL” instead of just “openssl”.

[apietersen]

OK I will try: “splunk cmd OpenSSL” instead of just “openssl”.

Thanks

[isoutamo]

Please use lowercase version for openssl. Autocorrect has meshed it 😞

[apietersen]

Thanks, I will do. 😀

[apietersen]

Our https certificate (web) is not expired. So what and where should I remove this old cert reference.
We only run Indexer, SH , licens Manager and Kvstore process....and nothing else

Please note: we run a small and single Splunk Enterprise instance on a Intel 16core , 64Gb mem, and 2Tb hardware under windows2019.

BTW How many certs do I need to run and manage in v9.1.0.1? 

Thanks

[isoutamo]

There are own certificates for web, splunkd and traffic between UFs and indexers. Mongod is using the same certificate as splunkd. You should check from server.conf where it is. 

[apietersen]

Hi,

looked in server.conf: (both test server + production)

Sorry , can not find any path or refence to certificates?

test-server:

[isoutamo]

You should use btool to check what and where those configurations are set. 

splunk btool server list --debug

[apietersen]

On my Test-server, I have copied a fresh server.conf from default directory and changed the license manager field. It is coming up and I can login but but still hangs on endless "loading..." when trying to change something (like restart)

Also Assist show an error:

[apietersen]

Update:

Yesterday we had a remote session (Zoom) with Splunk Support. No quick fix found. It needs to be investigated further:

Their observations:

------------------

- After upgrading the Splunk version to 9.1.0.1, the sendemail command is not working it is giving some errors on the production instance and test instance.

command="sendemail", The email domains of the recipients are not among those on the allowed domain list. while sending mail to: a.pietersen@.... 

command="sendemail", (530, b'5.7.57 Client not authenticated to send mail. [AS4P195CA0039.EURP195.PROD.OUTLOOK.COM 2023-07-25T10:35:23.523Z 08DB887EFF8B2458]', 'pietersen@i....) while sending mail to: a.pietersen@....

- The top of the screen showed a loading icon as I was reviewing the Server settings for the Email settings. Nothing can be done on the Email settings page.

- Additionally, we attempted to restart the UI by accessing the server controls. The loading icon at the top of the page appeared on the server control page as well, indicating the same problem.

- Additionally, the Splunk assist page is not working and displays the error message "Error Loading Splunk Assist."

- Even though we attempted to restart the search head using the CLI, the errors and issues persisted after the restart.

------------------

Thanks, hope they can find a remedy soon, because I am out of my option for now. 

[isoutamo]

Normally this  message means, that your smtp relay don’t allow you to send email with your from-domain. Are you sure that your M365 exchange has configured to relay those emails? 

[apietersen]

Hi
Both account belongs to me but have different domains. (and no trust-relation in place) Manual sending is no problem and use this daily. It was working for years now based on O365, it sopped after the upgarde. Besides no messages were received form MS that O365 have changed their relay rules. So for now I suspect something not OK in the Splunk 9.1.0.1 environment.

Thanks

[apietersen]

Update:

My main problem was not able to send critical Alerts and other message by email any more after upgradding to v9.1.0.1.

Finally I was able to solve this by copying the latest 'sendemail.py" from my backup of v9.0.5. into directory: ....\Splunk\etc\apps\search\bin .  After that I could send email as before.

Note: The "sendemail.py" that I copied from backup seems to be 2 lines longer than the new one installed by v9.1.0.1. Did not investigate it further, what the differences is. Busy to correct temp workarounds and made me and my users happy Splunking again. 🙂 Other issues are still open, but for now no showstoppers. Have send this info also to Splunk support but did not here back from them yet.

Thanks

[apietersen]

Update:

Last Friday Spiunk Support responded: problem with "sendemail" was identified as a bug and will be solved in next version.

Nb. Just noticed version v9.1.0.2 but there was no mention of/refering to this specific fix. I will wait for next version. Sendemail is still working after restoring the senemail.py form backup v9.0.5