Hi, We have the same issue here. Upgraded from Splunk Ent. v9.3.2 to V9.40 , running Windows 2019 server. The Kvstore process not running also effect on Splunk Secure Gateway (SSG/Splunk Mobile), Dastboard Studio, (and i think Edge Hub etc). 😞 Yes, looked in mongod.log and splunkd.log but not a bit wiser! See below some lines in my mongod.log : targetMinOS: Windows 7/Windows Server 2008 R2 - ??? this build only supports versions up to 4, and the file is version 5: - ??   2025-01-15T14:45:22.046Z I CONTROL [initandlisten] MongoDB starting : pid=2224 port=8191 dbpath=D:\Program Files\Splunk\var\lib\splunk\kvstore\mongo 64-bit host=Gozer2 2025-01-15T14:45:22.046Z I CONTROL [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2 2025-01-15T14:45:22.046Z I CONTROL [initandlisten] db version v4.2.24 2025-01-15T14:45:22.046Z I CONTROL [initandlisten] git version: 5e4ec1d24431fcdd28b579a024c5c801b8cde4e2 2025-01-15T14:45:22.046Z I CONTROL [initandlisten] allocator: tcmalloc 2025-01-15T14:45:22.046Z I CONTROL [initandlisten] modules: enterprise 2025-01-15T14:45:22.046Z I CONTROL [initandlisten] build environment: 2025-01-15T14:45:22.046Z I CONTROL [initandlisten] distmod: windows-64 2025-01-15T14:45:22.046Z I CONTROL [initandlisten] distarch: x86_64 2025-01-15T14:45:22.046Z I CONTROL [initandlisten] target_arch: x86_64 2025-01-15T14:45:22.047Z I CONTROL [initandlisten] options: { net: { bindIp: "0.0.0.0", port: 8191, tls: { CAFile: "D:\Program Files\Splunk\etc\auth\cacert.pem", allowConnectionsWithoutCertificates: true, allowInvalidCertificates: true, allowInvalidHostnames: true, certificateSelector: "subject=SplunkServerDefaultCert", disabledProtocols: "noTLS1_0,noTLS1_1", mode: "requireTLS", tlsCipherConfig: "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RS..." } }, replication: { oplogSizeMB: 200, replSet: "102D93C2-E5B9-4347-88CA-59FB829D92E1" }, security: { javascriptEnabled: false, keyFile: "D:\Program Files\Splunk\var\lib\splunk\kvstore\mongo\splunk.key" }, setParameter: { enableLocalhostAuthBypass: "0", oplogFetcherSteadyStateMaxFetcherRestarts: "0" }, storage: { dbPath: "D:\Program Files\Splunk\var\lib\splunk\kvstore\mongo", engine: "wiredTiger", wiredTiger: { engineConfig: { cacheSizeGB: 4.65 } } }, systemLog: { timeStampFormat: "iso8601-utc" } } 2025-01-15T14:45:22.048Z W NETWORK [initandlisten] sslCipherConfig parameter is not supported with Windows SChannel and is ignored. 2025-01-15T14:45:22.048Z W NETWORK [initandlisten] sslCipherConfig parameter is not supported with Windows SChannel and is ignored. 2025-01-15T14:45:22.049Z I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=4761M,cache_overflow=(file_max=0M),session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress], 2025-01-15T14:45:22.083Z E STORAGE [initandlisten] WiredTiger error (-31802) [1736952322:82769][2224:140709387064240], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error Raw: [1736952322:82769][2224:140709387064240], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error 2025-01-15T14:45:22.100Z E STORAGE [initandlisten] WiredTiger error (-31802) [1736952322:100690][2224:140709387064240], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error Raw: [1736952322:100690][2224:140709387064240], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error 2025-01-15T14:45:22.116Z E STORAGE [initandlisten] WiredTiger error (-31802) [1736952322:115624][2224:140709387064240], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error Raw: [1736952322:115624][2224:140709387064240], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error 2025-01-15T14:45:22.150Z E STORAGE [initandlisten] WiredTiger error (-31802) [1736952322:149476][2224:140709387064240], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error Raw: [1736952322:149476][2224:140709387064240], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error 2025-01-15T14:45:22.175Z E STORAGE [initandlisten] WiredTiger error (-31802) [1736952322:175362][2224:140709387064240], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error Raw: [1736952322:175362][2224:140709387064240], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error 2025-01-15T14:45:22.179Z W STORAGE [initandlisten] Failed to start up WiredTiger under any compatibility version. 2025-01-15T14:45:22.179Z F STORAGE [initandlisten] Reason: -31802: WT_ERROR: non-specific WiredTiger error 2025-01-15T14:45:22.179Z F - [initandlisten] Fatal Assertion 28595 at src\mongo\db\storage\wiredtiger\wiredtiger_kv_engine.cpp 928 2025-01-15T14:45:22.179Z F - [initandlisten] \n\n***aborting after fassert() failure\n\n   Some lines from my Slunkd.log: 01-15-2025 15:57:57.139 +0100 INFO TailReader [7248 tailreader0] - Batch input finished reading file='D:\Program Files\Splunk\var\spool\splunk\tracker.log' 01-15-2025 15:57:57.467 +0100 ERROR KVStorageProvider [5552 TcpChannelThread] - An error occurred during the last operation ('replSetGetStatus', domain: '15', code: '13053'): No suitable servers found (`serverSelectionTryOnce` set): [connection closed calling hello on 'gozer2:8191'] 01-15-2025 15:57:57.467 +0100 ERROR KVStoreAdminHandler [5552 TcpChannelThread] - An error occurred. 01-15-2025 15:58:03.592 +0100 ERROR KVStorageProvider [924 KVStoreUpgradeStartupThread] - An error occurred during the last operation ('replSetGetStatus', domain: '15', code: '13053'): No suitable servers found (`serverSelectionTryOnce` set): [connection closed calling hello on 'gozer2:8191'] 01-15-2025 15:58:10.645 +0100 ERROR KVStorageProvider [924 KVStoreUpgradeStartupThread] - An error occurred during the last operation ('replSetGetStatus', domain: '15', code: '13053'): No suitable servers found (`serverSelectionTryOnce` set): [connection closed calling hello on 'gozer2:8191'] 01-15-2025 15:58:17.723 +0100 ERROR KVStorageProvider [924 KVStoreUpgradeStartupThread] - An error occurred during the last operation ('replSetGetStatus', domain: '15', code: '13053'): No suitable servers found (`serverSelectionTryOnce` set): [connection closed calling hello on 'gozer2:8191'] 01-15-2025 15:58:24.745 +0100 WARN ExecProcessor [10156 ExecProcessor] - message from ""D:\Program Files\Splunk\bin\splunk-regmon.exe"" BundlesUtil - D:\Program Files\Splunk\etc\system\metadata\local.meta already exists but with different casing: D:\Program Files\splunk\etc\system\metadata\local.meta 01-15-2025 15:58:24.792 +0100 ERROR KVStorageProvider [924 KVStoreUpgradeStartupThread] - An error occurred during the last operation ('replSetGetStatus', domain: '15', code: '13053'): No suitable servers found (`serverSelectionTryOnce` set): [connection closed calling hello on 'gozer2:8191'] 01-15-2025 15:58:27.307 +0100 INFO TailReader [7248 tailreader0] - Batch input finished reading file='D:\Program Files\Splunk\var\spool\splunk\tracker.log' 01-15-2025 15:58:31.865 +0100 ERROR KVStorageProvider [924 KVStoreUpgradeStartupThread] - An error occurred during the last operation ('replSetGetStatus', domain: '15', code: '13053'): No suitable servers found (`serverSelectionTryOnce` set): [connection closed calling hello on 'gozer2:8191'] 01-15-2025 15:58:38.929 +0100 ERROR KVStorageProvider [924 KVStoreUpgradeStartupThread] - An error occurred during the last operation ('replSetGetStatus', domain: '15', code: '13053'): No suitable servers found (`serverSelectionTryOnce` set): [connection closed calling hello on 'gozer2:8191'] 01-15-2025 15:58:46.000 +0100 ERROR KVStorageProvider [924 KVStoreUpgradeStartupThread] - An error occurred during the last operation ('replSetGetStatus', domain: '15', code: '13053'): No suitable servers found (`serverSelectionTryOnce` set): [connection closed calling hello on 'gozer2:8191'] 01-15-2025 15:58:53.049 +0100 ERROR KVStorageProvider [924 KVStoreUpgradeStartupThread] - An error occurred during the last operation ('replSetGetStatus', domain: '15', code: '13053'): No suitable servers found (`serverSelectionTryOnce` set): [connection closed calling hello on 'gozer2:8191'] 01-15-2025 15:58:56.617 +0100 INFO TailReader [7248 tailreader0] - Batch input finished reading file='D:\Program Files\Splunk\var\spool\splunk\tracker.log' 01-15-2025 15:59:00.117 +0100 ERROR KVStorageProvider [924 KVStoreUpgradeStartupThread] - An error occurred during the last operation ('replSetGetStatus', domain: '15', code: '13053'): No suitable servers found (`serverSelectionTryOnce` set): [connection closed calling hello on 'gozer2:8191'] 01-15-2025 15:59:01.460 +0100 ERROR KVStorageProvider [5608 TcpChannelThread] - An error occurred during the last operation ('replSetGetStatus', domain: '15', code: '13053'): No suitable servers found (`serverSelectionTryOnce` set): [connection closed calling hello on 'gozer2:8191'] 01-15-2025 15:59:01.460 +0100 ERROR KVStoreAdminHandler [5608 TcpChannelThread] - An error occurred. ... View more

Summarize and finishing this post: After installing Splunk Ent v9.3.2 the initial problems were solved. After that it appears that Splunk Secure Gateways did not connect anymore (Splunk Mobile) Had contact with Splunk about this. After some back and forth copying of backup directories and running repair, chaning app.cong files etc, SSG became connected again. The main reason why I want to upgrade were some new features that were mentioned to be available in Dashboard Studio. Unfortunately, now the present Dashboard Studio (version 1.15.3 under v9.3.2)  has some other issues, like: eg background color shows black instead of transparent (on mobile/table), big-font size 22+ text is unreadable small in Markdown (on mobile/tablet), using title color is  neither working as I expected... Nb. besides the fact that also the new tab feature was already noted not to work in this version including next version  v9.4.0. so I understand. So here I close this post and will create a new post next year about Dashboard Studio. In the meantime waiting for an update to v9.4.X  @all, have a great new year 2025! AshleyP ... View more

I was able to install v9.3.2 without SSLEAY32.dll and LIBEAY32 .dll errors  But now our Splunk Secure Gateways has stopped working 😞 Solving one issue and auto introduced to another issue 😞 Unable to initialize modular input "ssg_subscription_modular_input" defined in the app "splunk_secure_gateway": Introspecting scheme=ssg_subscription_modular_input: script running failed (exited with code 1)..11-12-2024, 10:03:22   websocket:   Error in 'checkssgmobilewss' command: External search command exited unexpectedly with non-zero error code 1. HTTPS(Sync): OK HTTPS( Async): Error in 'checkssgmobileasync' command: External search command exited unexpectedly with non-zero error code 1. Does somebody has a suggestion what to do next? I am pretty much lost here, for the moment.  AshleyP ... View more

I have found this info sofar: https://splunk.my.site.com/customer/s/article/The-code-execution-cannot-proceed-because-LIBEAY32-dll-was-not-found-Reinstalling-the-program-may-fix-this-problem ... View more

Unfortunately , skipping 9.3.0 and tried to install v9.3.1, same result: I suppose, I need to create a ticket 😞   ... View more

Hi SanjayReddy, Thanks for your response. Looked in de logfile directory but could not find any file (or content in any files) that was referring to the upgrade !? Ok, I will try to skip 9.3.0 and see what happens, in the coming days. 😨 I will post any results here. Thanks 👍 ... View more

This morning I did a repair on the v9.2.4 (with b9.2.4 of course) and tried to upgrade to v9.3.0 again Validating install messages during the process, so nothing to worry about I thought?? At the end again: "code execution script cannot proceed SSLEAY32.dll was not found. Reinstalling the program may fix this problem" - After this error it automatically starts a rollback, luckily, but I'm stuck on v9.2.4 now. Questions: What is a validating installation message, what is checked and what is not? Can I view/look into this code somewhere to check if it points to the right directory Is there a log file of the upgrade??? Has anyone had this problem before and/or does anyone know what I should do next? I am considering creating a support ticket. Have done several upgrades over the years to our Splunk Enterprise platform but never experienced this. Any suggestion or tip is welcome. ... View more

Update: Had a session today with O365 support about message in python.log: (posted before) SendAsDenied; ticket@eremote.nl not allowed to send as Splunk_eRemote@uBDC01;" Answer: As discussed on the call, the bounceback (NDR report) you received shows that your office 365 account (ticket@eremote.nl) is not allowed to send email as Splunk_eRemote@uBDC01. This shows that there is a setting in Splunk that is preventing this action. And like I mentioned on the call, from Microsoft 365 perspective, email relayed is allowed and that is why you can send normal email from the application, and it delivers. De 'Splunk_eRemote@uBDC01'  comes from the SMTP-server settings in Splunk: Note: 1) We use O365 and some 4-5 months ago we stepped over from a personal account to a shared account (but did not notice the defect in the Alerting_email_function, just last week :-(. As Sendemail was working in Dasboards and in SPL code as expected. 2) We now use a shared mailbox under O365, which means you can not use any Alias. Before we could. 3) Using my personal account (with Alias's) is 2FA and is not possible according to O365 support. 4) Last week tried many things -including different fields A en B, without succes. Also tested it with use of a GMAIL account, no succes either. 5) Yesterday (Sunday) rebooted our W2029 server as part of our weekly maintenance schedule,  Today I found out: Note: Field A (username) must be same as SendAS (in the past we used only the word "Splunk" in field B) Also note that keeping the B feidl empty it does a discovery and comes up with "Splunk@uBDC01"  so it appears.  Now it is working again (in test) with field-A = field-B! (what is de use? apperantly only working in combination with personal-account and proper Alias, I conclude) I will close this post now and thank you for the repsonse PickeRick :-)! AshleyP ... View more

I have already send a support request to MS Office365. Hope that they can give a better or more detailed reason then "SendAsDenied; ticket@eremote.nl not allowed to send as Splunk_eRemote@uBDC01;" In the mean time I will follow also my Splunk support-case route... Will post my findings here, later.... ... View more

Yes it does when using: | makeresults | sendemail to="aaa@ccc", from="bbb@eee.nl", subject="Testing Email from Splunk / from: uBDC01", use_ssl=false, use_tls=true or as form in a Dashboard we use to create a ticket for our customer. Only alerting has stop working recently? BTW: we use OFFICE365 so this should be occuring elsewhere too, do not you think?. And no errors-essaging is showing in Splunk (as user or as admin) when this happens. So you suggest to look in the O365 log of the exchange environment? ... View more

Hi PickleRick, Thanks,  1) Yes, I have tried many thing, also by using the opposite condition: alert_condition = search mincount < 7 2) In all the logs I have checked (including sendemail.py, which does not log anything in my opinion) and searched, it is/was successful, status: success, not skipped or otherwise found to be delayed or ??? 3) example spl:  index=_* AND alert_actions="email" | stats count by status Output: 42 Success over last 24h  02-17-2024 12:12:17.264 +0100 INFO SavedSplunker - savedsearch_id="nobody;search;Alert_trigger_1v1", search_type="scheduled", search_streaming=0, user="admin", app="search", savedsearch_name="Alert_trigger_1v1", priority=default, status=success, digest_mode=0, durable_cursor=0, scheduled_time=1708168200, window_time=0, dispatch_time=1708168333, run_time=0.142, result_count=1, alert_actions="email", sid="scheduler__admin__search__RMD5ea1ed26b5154d33f_at_1708168200_47", suppressed=0, fired=1, skipped=0, action_time_ms=3528, thread_id="AlertNotifierWorker-0", message="", workload_pool=""   So no error message to be found sofar??  What do I miss here. Although I also searched in python.log and I found part of many messages: 2024-02-17 12:32:17,265 +0100 ERROR sendemail:572 - (554, b'5.2.252 SendAsDenied; ticket@eremote.nl not allowed to send as Splunk_eRemote@uBDC01; STOREDRV.Submission.Exception:SendAsDeniedException.MapiExceptionSendAsDenied; Failed to process message due to a permanent exception with message [BeginDiagnosticData]Cannot submit message. ....>> Splunk_eRemote@uBDC01;  Splunk_eRemote is a textfield in SMPT-settings. Python code seems to append "@uBDC01" to it (is the hostname of my testserver ???) I  can not make any sense out of it (same setting used in past 6 year?!) on our production server Again: Sendemail is working fine, in dashboard-search (form) and when using it for testing by running manual SPL code in searches Any thoughts? regards AshleyP ... View more

Did you upgrade to v9.1.2 already?  If so, I suggest you create a ticket at support. The replacing was a temp work-around solution for us from v9.1.0.2. On our test server I was curious if this work-around would still work on v9.1.1 - It did not ! So I decide to wait for the release of v9.1.2.  After the release I first tested on our test server and had no problem any more with sending email.  After some days we upgraded to V9.1.2 on our production machine. Nb. Both our servers are running Windows 2019, and now both are on Splunk Enterprise v9.1.2 without problems so far. ... View more

Completely ridiculous, at Nov-2023 on Splunk Enterprise v91.1.1 still an issue installing Python for Scientific Computing (for Windows 64-bit), even after change setting upload limit in web.conf as documented # Default is to allow files up to 500MB to be uploaded to the server # change this in local/web.conf if necessary # max_upload_size = 500. max_upload_size = 5000. There was an error processing the upload.Error during app install: failed to extract app from C:\Windows\TEMP\tmpp0s51vp5 to D:\Program Files\Splunk\var\run\splunk\bundle_tmp\f8e20425d3b382c7: The system cannot find the path specified. What do I miss here, how much must be this size be to , what units, or where are the brains of splunk. Splunk Enterprise after v9. 1.x is full of **bleep**.  ... View more

response from support / development: (last week) and now testing, worked and looks good sofar temporary workaround: --=-=--=-=-=-==-=-sendmail.py-=--=-=-=-=-- Do you have a test instance to check the one parameter? We got an update from the developer team, to check the below parameter, open the file SPLUNK_HOME/etc/apps/search/bin/sendemail.py and you would find (approx line number 1571): clear_password = cli_common.decrypt(encrypted_password, setEnv=True) The setEnv flag needs to be modified from True to False, restart Splunk, and then check if the problem is resolved. Note: Try with your test instance. As the setEnv flag needs to be modified from True to False will be considered a temporary workaround for this issue. In the next release version 9.1.2, It will be fixed completely. -=-=-=-=-=---=-icon-=-=-=-=--= Additionally, the "loading" icon got stuck issue also will be fixed in version 9.1.2. -=-=--=-= regards AP  @wskinner @isoutamo  ... View more

Hi, this worked for me, in file ...etc\system\local\web_feature.conf: [feature:dashboards_csp] enable_dashboards_redirection_restriction = false ... View more

Update: support reported the email issue has changed prority from P2 to P3 All other issues mentioned in this post: no solution found yet ... View more

Update 09-aug: the second remote zoom on monday 7-aug meeting did not results in any solution sofar unfortunately. All issues are still under investigation by support and developement, so I understand. thanks ... View more

Looking back: v9.0.4.x no issues noticed v9.0.5.x noticed KVStore issues  v9.1.0.1 noticed all sort of issue including not working 'sendemail'  (upgraded as I had hoped/assumed to be fixed) v9.1.0.2 still the same issues + new strange behaviour : only reading 3 columns from any csv file (upgraded as I hoped/assumed to be fixed) Nb. Have a zoom meeting with Splunk support later this afternoon thanks Reminder: running on Windows 2019 standard,  Xeon CPU, 16 cores, 64Gb , 2Tb ssd ... View more

After the upgrade to v9.1.0.2, last weekend - hoping and assuming some issues were adressed and fixed, as noted in this post, now another issue arised, which make our production-instance even more unreliable. When using an siimple '| inputlookup email.csv' search it only show 3 columns !  On our fresh build test-server with instance v91.0.2 the same serahc produce the correct number of columns. Has somebody experinced this same issue or have a glue why or have a tip. Or what do I miss here? By the way. the sendemail did not work either after upgrade to v91.0.2, after restoring an sendemail.py file from backup v9.0.4 (as I described in this post  earlier sendemail was working again, for now. But what is next! 😞  Nb. Sorry, I really think Splunk should pay more attention to testing and quality control before releasing new versions.  thanks ... View more

Update,  finally I succeed by using the subsearch as was mentioned before + creating 3x6=18 daily scheduled reports. (6 machines)       Dahboard to split in 2 tables panel... Nb. Unfortunately I was not able to make a historical search all-in-one! Thanks ... View more

update: after upgrade to Splunk Enterprise v9.1.0.2 , "sendemail" issue still exist.   ... View more

Update: Last Friday Spiunk Support responded: problem with "sendemail" was identified as a bug and will be solved in next version. Nb. Just noticed version v9.1.0.2 but there was no mention of/refering to this specific fix. I will wait for next version. Sendemail is still working after restoring the senemail.py form backup v9.0.5 ... View more