What is happening in Splunk Enterprise V9.1.0.1 ?

[apietersen]

RE: Case #3270697 After upgrade to 9.1.01 not able to send emails eg. of critical alerts! [ ref:_00D409oyL._5005a2bGRKI:ref ]

After upgrade to v9.1.0.1 Splunk Enterprise, (single instance), last weekend (15 Juli 2023) + changing admin password as was suggested by Assist (which throws an error now !?)

1) Message when using sendemail:

Smpt setting: O365

Checked the login on O365, ofcourse

2) Assist stopped running???

3) Also:

3a

3b

4) new GUI / layout ?

5) Annoying and not working “Don’t show this again” message on every page. Just stepping to another dashboard on the same server/domain ?? 

6) endless waiting:

What is next?

Anyone else suffering from the same issues?

[apietersen · Accepted solution]

Last week v9.1.2 has been released. (6 nov 2023, I think it was)

After installing this version on my test instance (v9.1.1) everytyhing seems to work again including sendemail - no issues found. Great! 👍

After installing this version several days later on our production instance (9.1.0.2) also sendemail was working fine again. Great! 👍

NB. after that I was also able to fix all other issues on our production instance as mentioned before in this post, like: kvstore, secure gateway etc Great! 👍

Many thanks to support- and development team. I am now happy splunking again! 👍👍 😀

I hereby close this post.

[nioann]

I had the same issue with sending emails. I was able to resolve it by replacing the sendemail.py file in $SPLUNKetc\apps\search\bin, with an older version of sendemail.py. I still have the issue with endless loading on settings pages. How were you able to resolve this?

[apietersen]

Did you upgrade to v9.1.2 already?  If so, I suggest you create a ticket at support.

The replacing was a temp work-around solution for us from v9.1.0.2.

On our test server I was curious if this work-around would still work on v9.1.1 - It did not !
So I decide to wait for the release of v9.1.2. 

After the release I first tested on our test server and had no problem any more with sending email. 
After some days we upgraded to V9.1.2 on our production machine.

Nb. Both our servers are running Windows 2019, and now both are on Splunk Enterprise v9.1.2 without problems so far.

[nioann]

Will do. Thanks for the response.

[apietersen]

Update: support reported the email issue has changed prority from P2 to P3
All other issues mentioned in this post: no solution found yet

[wskinner]

Did you ever get a resolution? @apietersen ? Upgraded to 9.1.1 from 9.0.5 and saw that email alerts ceased.  I'm getting a similar message to you

[apietersen]

response from support / development: (last week) and now testing,
worked and looks good sofar

temporary workaround:

--=-=--=-=-=-==-=-sendmail.py-=--=-=-=-=--
Do you have a test instance to check the one parameter?

We got an update from the developer team, to check the below parameter,
open the file SPLUNK_HOME/etc/apps/search/bin/sendemail.py
and you would find (approx line number 1571):

clear_password = cli_common.decrypt(encrypted_password, setEnv=True)
The setEnv flag needs to be modified from True to False, restart Splunk, and then check if the problem is resolved.

Note: Try with your test instance.

As the setEnv flag needs to be modified from True to False will be considered a temporary workaround for this issue. In the next release version 9.1.2, It will be fixed completely.

-=-=-=-=-=---=-icon-=-=-=-=--=

Additionally, the "loading" icon got stuck issue also will be fixed in version 9.1.2.

-=-=--=-=

regards AP 
@wskinner @isoutamo 

[wskinner]

this worked - thanks.  lol I was looking for the differences between the previous versions of sendemail.  I was hoping that 9.1.2 would have gotten deployed sooner than later, but this will work until then.

[O815163]

replacing sendemail.py worked for me as well.

linking related post: https://community.splunk.com/t5/Other-Usage/Why-is-Splunk-send-email-function-not-working-version-9-...

[apietersen]

Looking back:
v9.0.4.x no issues noticed
v9.0.5.x noticed KVStore issues 
v9.1.0.1 noticed all sort of issue including not working 'sendemail'  (upgraded as I had hoped/assumed to be fixed)
v9.1.0.2 still the same issues + new strange behaviour : only reading 3 columns from any csv file (upgraded as I hoped/assumed to be fixed)

Nb. Have a zoom meeting with Splunk support later this afternoon

thanks

Reminder: running on Windows 2019 standard,  Xeon CPU, 16 cores, 64Gb , 2Tb ssd

[apietersen]

Update 09-aug: the second remote zoom on monday 7-aug meeting did not results in any solution sofar unfortunately. All issues are still under investigation by support and developement, so I understand.

thanks

[apietersen]

After the upgrade to v9.1.0.2, last weekend - hoping and assuming some issues were adressed and fixed, as noted in this post, now another issue arised, which make our production-instance even more unreliable.

When using an siimple '| inputlookup email.csv' search it only show 3 columns !  On our fresh build test-server with instance v91.0.2 the same serahc produce the correct number of columns.

Has somebody experinced this same issue or have a glue why or have a tip. Or what do I miss here?

By the way. the sendemail did not work either after upgrade to v91.0.2, after restoring an sendemail.py file from backup v9.0.4 (as I described in this post  earlier sendemail was working again, for now. But what is next! 😞 

Nb. Sorry, I really think Splunk should pay more attention to testing and quality control before releasing new versions. 

thanks

[apietersen]

update: after upgrade to Splunk Enterprise v9.1.0.2 , "sendemail" issue still exist.  

[apietersen]

Update:

SORRY, AT THE MOMENT NOT MUCH HAPPY SPLUNKING HERE

WE RUN A 2nd Splunk Enterprise server (FOR TEST/DEV etc) and most issue mentioned below shows on both machines….

Nb. A week before we upgraded our production machine we installed v9.0.1 on our test-server and we only noticed point 5 and 6 but were not alarmed by the things that would come later after we decided to go forward. It came to our attention after we did not see any Alerts and other emails were send.

After upgrade to v9.1.0.1 Splunk Enterprise, (single instance), last weekend (15 Juli 2023) + changing admin password as was suggested by Assist (which throws an error today / now !?)

1) Message when using sendemail:

Smpt setting: O365

Checked the login on O365, ofcourse

ITEM 1 IS STILL NOT SOLVED

2) Assist stopped running???

ITEM 2 IS STILL NOT SOLVED

3) Also issue with:

3a

3b -after restart:

 ITEM  3a and 3b IS STILL NOT SOLVED

4) new GUI / layout ?

 ITEM  4 IS NOT ANSWERED YET

5) Annoying and not working “Don’t show this again” message on every page. Just stepping to another dashboard on the same server/domain ?? 

 ITEM 5 HAS BEEN SOLVED

6) endless waiting:

 ITEM  6 IS STILL NOT SOLVED – ANY CONFIG CHANGED IN WEB GUI IS NOT WORKING AND ANY CONFIG PAGE KEEPS HANGING. "Loading"

[apietersen]

Thanks I will check...

[isoutamo]

Hi

based on that kvstore/mongodb message I suspect that mongoldb is not running? You could check it's status by 

splunk show kvstore-status --verbose

If it's not up and running you you'd found there reason from $SPLUNK_HOME/var/log/splunk/mongod.log  

Probably most common reason for "not running kvstore/mongod" is expired TLS certificate.

r. Ismo

[apietersen]

Hi isoutamo

I ran the kwstore diag, as admin: 

• not sure if this is TLS related?
• also do not understand why I should have TLS enabled to acces to something running on the same single server/Splunk Enterprise instance?
• where can I disable this in conf file?
• or how to renwe cert??

thanks

[apietersen]

mongod.log shows:

2023-07-21T21:14:54.507+0200 W CONTROL [main] Option: sslMode is deprecated. Please use tlsMode instead.
2023-07-21T21:14:54.508+0200 W CONTROL [main] Option: sslCipherConfig is deprecated. Please use tlsCipherConfig instead.
2023-07-21T21:14:54.508+0200 W CONTROL [main] Option: sslAllowConnectionsWithoutCertificates is deprecated. Please use tlsAllowConnectionsWithoutCertificates instead.
2023-07-21T21:14:54.508+0200 W CONTROL [main] Option: sslAllowInvalidHostnames is deprecated. Please use tlsAllowInvalidHostnames instead.
2023-07-21T21:14:54.508+0200 W CONTROL [main] Option: sslAllowInvalidCertificates is deprecated. Please use tlsAllowInvalidCertificates instead.
2023-07-21T21:14:54.508+0200 W CONTROL [main] Option: sslCertificateSelector is deprecated. Please use tlsCertificateSelector instead.
2023-07-21T19:14:54.513Z W CONTROL [main] net.tls.tlsCipherConfig is deprecated. It will be removed in a future release.
2023-07-21T19:14:54.528Z W NETWORK [main] sslCipherConfig parameter is not supported with Windows SChannel and is ignored.
2023-07-21T19:14:54.529Z W NETWORK [main] sslCipherConfig parameter is not supported with Windows SChannel and is ignored.
2023-07-21T19:14:54.529Z F NETWORK [main] The provided SSL certificate is expired or not yet valid.
2023-07-21T19:14:54.529Z F - [main] Fatal Assertion 50755 at src\mongo\util\net\ssl_manager_windows.cpp 1609
2023-07-21T19:14:54.529Z F - [main] \n\n***aborting after fassert() failure\n\n

• Where and what can I change to get it working again or ignore these TLS SSL and/expired messages?
• Where to renew these "indoor"  TLS certs  (self-signed cert I assume and hope) ?

Splunk.log shows:

07-22-2023 11:05:55.181 +0200 WARN  SearchOperator:kv [13064 SchedulerThread] - Could not find a transform named REPORT-USPS-OFFLINE-CSV
07-22-2023 11:05:55.181 +0200 WARN  SearchOperator:kv [13064 SchedulerThread] - Could not find a transform named REPORT-Camera1
07-22-2023 11:05:55.181 +0200 WARN  SearchOperator:kv [13064 SchedulerThread] - Could not find a transform named REPORT-usps-off4
07-22-2023 11:05:55.181 +0200 WARN  SearchOperator:kv [13064 SchedulerThread] - Could not find a transform named REPORT-USPS-OFFLINE-CSV
07-22-2023 11:05:55.181 +0200 WARN  SearchOperator:kv [13064 SchedulerThread] - Could not find a transform named REPORT-usps-off4
07-22-2023 11:05:55.181 +0200 WARN  SearchOperator:kv [13064 SchedulerThread] - Could not find a transform named REPORT-USPS-OFFLINE-CSV
07-22-2023 11:05:55.181 +0200 WARN  SearchOperator:kv [13064 SchedulerThread] - Could not find a transform named REPORT-Camera1
07-22-2023 11:05:55.213 +0200 INFO  NoahSearchPeerFetcher [13064 SchedulerThread] - Fetch requested. sid=scheduler__nobody__search__RMD5883a9bd5121d9759_at_1690016700_59 use_cache=1
07-22-2023 11:06:00.268 +0200 INFO  ExecProcessor [10852 ExecProcessor] - setting reschedule_ms=59732, for command="D:\Program Files\Splunk\bin\Python3.exe" "D:\Program Files\Splunk\etc\apps\search\bin\quarantine_files.py"

Thanks

[isoutamo]

As it said your tls/ssl cert has expired you need to renew it. 
If you are using Splunk’s default certs (you shouldn’t), just remove/rename old certificate file and restart splunk. That will generate a new one. If you are using official from some cert authority or your own, then you must get a new from there and replace current with that.

You could check that e.g. with command 

openssl x509 -in mycert.pem -text -noout

This told the validity and who has guaranteed it. 

[apietersen]

sorry, both on test-server and on production server shows: