topic Re: What is happening in Splunk Enterprise V9.1.0.1 ? in Splunk Enterprise

What is happening in Splunk Enterprise V9.1.0.1 ?

apietersen — Thu, 20 Jul 2023 14:30:58 GMT

RE: Case #3270697 After upgrade to 9.1.01 not able to send emails eg. of critical alerts! [ ref:_00D409oyL._5005a2bGRKI:ref ]

After upgrade to v9.1.0.1 Splunk Enterprise, (single instance), last weekend (15 Juli 2023) + changing admin password as was suggested by Assist (which throws an error now !?)

1) Message when using sendemail:

Smpt setting: O365

Checked the login on O365, ofcourse

2) Assist stopped running???

3) Also:

3a

4) new GUI / layout ?

5) Annoying and not working “Don’t show this again” message on every page. Just stepping to another dashboard on the same server/domain ??

6) endless waiting:

What is next?

Anyone else suffering from the same issues?

Re: What is happening in Splunk Enterprise V9.1.0.1 ?

isoutamo — Fri, 21 Jul 2023 11:22:37 GMT

based on that kvstore/mongodb message I suspect that mongoldb is not running? You could check it's status by

splunk show kvstore-status --verbose

If it's not up and running you you'd found there reason from $SPLUNK_HOME/var/log/splunk/mongod.log

Probably most common reason for "not running kvstore/mongod" is expired TLS certificate.

r. Ismo

Re: What is happening in Splunk Enterprise V9.1.0.1 ?

apietersen — Fri, 21 Jul 2023 15:14:17 GMT

Thanks I will check...

Re: What is happening in Splunk Enterprise V9.1.0.1 ?

apietersen — Sat, 22 Jul 2023 08:57:58 GMT

Hi isoutamo

I ran the kwstore diag, as admin:

not sure if this is TLS related?
also do not understand why I should have TLS enabled to acces to something running on the same single server/Splunk Enterprise instance?
where can I disable this in conf file?
or how to renwe cert??

thanks

Re: What is happening in Splunk Enterprise V9.1.0.1 ?

apietersen — Sat, 22 Jul 2023 09:09:41 GMT

mongod.log shows:

2023-07-21T21:14:54.507+0200 W CONTROL [main] Option: sslMode is deprecated. Please use tlsMode instead. 2023-07-21T21:14:54.508+0200 W CONTROL [main] Option: sslCipherConfig is deprecated. Please use tlsCipherConfig instead. 2023-07-21T21:14:54.508+0200 W CONTROL [main] Option: sslAllowConnectionsWithoutCertificates is deprecated. Please use tlsAllowConnectionsWithoutCertificates instead. 2023-07-21T21:14:54.508+0200 W CONTROL [main] Option: sslAllowInvalidHostnames is deprecated. Please use tlsAllowInvalidHostnames instead. 2023-07-21T21:14:54.508+0200 W CONTROL [main] Option: sslAllowInvalidCertificates is deprecated. Please use tlsAllowInvalidCertificates instead. 2023-07-21T21:14:54.508+0200 W CONTROL [main] Option: sslCertificateSelector is deprecated. Please use tlsCertificateSelector instead. 2023-07-21T19:14:54.513Z W CONTROL [main] net.tls.tlsCipherConfig is deprecated. It will be removed in a future release. 2023-07-21T19:14:54.528Z W NETWORK [main] sslCipherConfig parameter is not supported with Windows SChannel and is ignored. 2023-07-21T19:14:54.529Z W NETWORK [main] sslCipherConfig parameter is not supported with Windows SChannel and is ignored. 2023-07-21T19:14:54.529Z F NETWORK [main] The provided SSL certificate is expired or not yet valid. 2023-07-21T19:14:54.529Z F - [main] Fatal Assertion 50755 at src\mongo\util\net\ssl_manager_windows.cpp 1609 2023-07-21T19:14:54.529Z F - [main] \n\n***aborting after fassert() failure\n\n

Where and what can I change to get it working again or ignore these TLS SSL and/expired messages?
Where to renew these "indoor" TLS certs (self-signed cert I assume and hope) ?

Splunk.log shows:

07-22-2023 11:05:55.181 +0200 WARN SearchOperator:kv [13064 SchedulerThread] - Could not find a transform named REPORT-USPS-OFFLINE-CSV 07-22-2023 11:05:55.181 +0200 WARN SearchOperator:kv [13064 SchedulerThread] - Could not find a transform named REPORT-Camera1 07-22-2023 11:05:55.181 +0200 WARN SearchOperator:kv [13064 SchedulerThread] - Could not find a transform named REPORT-usps-off4 07-22-2023 11:05:55.181 +0200 WARN SearchOperator:kv [13064 SchedulerThread] - Could not find a transform named REPORT-USPS-OFFLINE-CSV 07-22-2023 11:05:55.181 +0200 WARN SearchOperator:kv [13064 SchedulerThread] - Could not find a transform named REPORT-usps-off4 07-22-2023 11:05:55.181 +0200 WARN SearchOperator:kv [13064 SchedulerThread] - Could not find a transform named REPORT-USPS-OFFLINE-CSV 07-22-2023 11:05:55.181 +0200 WARN SearchOperator:kv [13064 SchedulerThread] - Could not find a transform named REPORT-Camera1 07-22-2023 11:05:55.213 +0200 INFO NoahSearchPeerFetcher [13064 SchedulerThread] - Fetch requested. sid=scheduler__nobody__search__RMD5883a9bd5121d9759_at_1690016700_59 use_cache=1 07-22-2023 11:06:00.268 +0200 INFO ExecProcessor [10852 ExecProcessor] - setting reschedule_ms=59732, for command="D:\Program Files\Splunk\bin\Python3.exe" "D:\Program Files\Splunk\etc\apps\search\bin\quarantine_files.py"

Thanks

Re: What is happening in Splunk Enterprise V9.1.0.1 ?

isoutamo — Sat, 22 Jul 2023 08:59:41 GMT

As it said your tls/ssl cert has expired you need to renew it.
If you are using Splunk’s default certs (you shouldn’t), just remove/rename old certificate file and restart splunk. That will generate a new one. If you are using official from some cert authority or your own, then you must get a new from there and replace current with that.

You could check that e.g. with command

openssl x509 -in mycert.pem -text -noout

This told the validity and who has guaranteed it.

Re: What is happening in Splunk Enterprise V9.1.0.1 ?

apietersen — Mon, 24 Jul 2023 07:19:37 GMT

Our https certificate (web) is not expired. So what and where should I remove this old cert reference.
We only run Indexer, SH , licens Manager and Kvstore process....and nothing else

Please note: we run a small and single Splunk Enterprise instance on a Intel 16core , 64Gb mem, and 2Tb hardware under windows2019.

BTW How many certs do I need to run and manage in v9.1.0.1?

Thanks

Re: What is happening in Splunk Enterprise V9.1.0.1 ?

apietersen — Mon, 24 Jul 2023 07:43:06 GMT

Update:

SORRY, AT THE MOMENT NOT MUCH HAPPY SPLUNKING HERE

WE RUN A 2nd Splunk Enterprise server (FOR TEST/DEV etc) and most issue mentioned below shows on both machines….

Nb. A week before we upgraded our production machine we installed v9.0.1 on our test-server and we only noticed point 5 and 6 but were not alarmed by the things that would come later after we decided to go forward. It came to our attention after we did not see any Alerts and other emails were send.

After upgrade to v9.1.0.1 Splunk Enterprise, (single instance), last weekend (15 Juli 2023) + changing admin password as was suggested by Assist (which throws an error today / now !?)

1) Message when using sendemail:

Smpt setting: O365

Checked the login on O365, ofcourse

ITEM 1 IS STILL NOT SOLVED

2) Assist stopped running???

ITEM 2 IS STILL NOT SOLVED

3) Also issue with:

3a

3b -after restart:

ITEM 3a and 3b IS STILL NOT SOLVED

4) new GUI / layout ?

ITEM 4 IS NOT ANSWERED YET

5) Annoying and not working “Don’t show this again” message on every page. Just stepping to another dashboard on the same server/domain ??

ITEM 5 HAS BEEN SOLVED

6) endless waiting:

ITEM 6 IS STILL NOT SOLVED – ANY CONFIG CHANGED IN WEB GUI IS NOT WORKING AND ANY CONFIG PAGE KEEPS HANGING. "Loading"

Re: What is happening in Splunk Enterprise V9.1.0.1 ?

isoutamo — Mon, 24 Jul 2023 07:54:01 GMT

There are own certificates for web, splunkd and traffic between UFs and indexers. Mongod is using the same certificate as splunkd. You should check from server.conf where it is.

Re: What is happening in Splunk Enterprise V9.1.0.1 ?

apietersen — Mon, 24 Jul 2023 08:37:39 GMT

Hi,

looked in server.conf: (both test server + production)

Sorry , can not find any path or refence to certificates?

test-server:

Re: What is happening in Splunk Enterprise V9.1.0.1 ?

apietersen — Mon, 24 Jul 2023 14:04:15 GMT

On my Test-server, I have copied a fresh server.conf from default directory and changed the license manager field. It is coming up and I can login but but still hangs on endless "loading..." when trying to change something (like restart)

Also Assist show an error:

Re: What is happening in Splunk Enterprise V9.1.0.1 ?

apietersen — Mon, 24 Jul 2023 14:21:45 GMT

sorry, both on test-server and on production server shows:

Re: What is happening in Splunk Enterprise V9.1.0.1 ?

isoutamo — Mon, 24 Jul 2023 14:53:22 GMT

You should use btool to check what and where those configurations are set.

splunk btool server list --debug

Re: What is happening in Splunk Enterprise V9.1.0.1 ?

apietersen — Wed, 26 Jul 2023 05:43:05 GMT

Update:

Yesterday we had a remote session (Zoom) with Splunk Support. No quick fix found. It needs to be investigated further:

Their observations:

------------------

- After upgrading the Splunk version to 9.1.0.1, the sendemail command is not working it is giving some errors on the production instance and test instance.

command="sendemail", The email domains of the recipients are not among those on the allowed domain list. while sending mail to: a.pietersen@....

command="sendemail", (530, b'5.7.57 Client not authenticated to send mail. [AS4P195CA0039.EURP195.PROD.OUTLOOK.COM 2023-07-25T10:35:23.523Z 08DB887EFF8B2458]', 'pietersen@i....) while sending mail to: a.pietersen@....

- The top of the screen showed a loading icon as I was reviewing the Server settings for the Email settings. Nothing can be done on the Email settings page.

- Additionally, we attempted to restart the UI by accessing the server controls. The loading icon at the top of the page appeared on the server control page as well, indicating the same problem.

- Additionally, the Splunk assist page is not working and displays the error message "Error Loading Splunk Assist."

- Even though we attempted to restart the search head using the CLI, the errors and issues persisted after the restart.

------------------

Thanks, hope they can find a remedy soon, because I am out of my option for now.

Re: What is happening in Splunk Enterprise V9.1.0.1 ?

isoutamo — Wed, 26 Jul 2023 09:40:13 GMT

Normally this message means, that your smtp relay don’t allow you to send email with your from-domain. Are you sure that your M365 exchange has configured to relay those emails?

Re: What is happening in Splunk Enterprise V9.1.0.1 ?

isoutamo — Wed, 26 Jul 2023 09:43:10 GMT

You should replace mycert.pem with your own cert file. That name you could see by previous btool. Use also full path to this file. Sometimes it’s also needed to add “splunk cmd OpenSSL” instead of just “openssl”.

Re: What is happening in Splunk Enterprise V9.1.0.1 ?

apietersen — Wed, 26 Jul 2023 10:10:33 GMT

Hi
Both account belongs to me but have different domains. (and no trust-relation in place) Manual sending is no problem and use this daily. It was working for years now based on O365, it sopped after the upgarde. Besides no messages were received form MS that O365 have changed their relay rules. So for now I suspect something not OK in the Splunk 9.1.0.1 environment.

Thanks

Re: What is happening in Splunk Enterprise V9.1.0.1 ?

apietersen — Wed, 26 Jul 2023 10:13:34 GMT

OK I will try: “splunk cmd OpenSSL” instead of just “openssl”.

Thanks

Re: What is happening in Splunk Enterprise V9.1.0.1 ?

isoutamo — Wed, 26 Jul 2023 10:39:21 GMT

Please use lowercase version for openssl. Autocorrect has meshed it 😞

Re: What is happening in Splunk Enterprise V9.1.0.1 ?

apietersen — Wed, 26 Jul 2023 10:41:36 GMT

Thanks, I will do. 😀