Update 09-aug: the second remote zoom on monday 7-aug meeting did not results in any solution sofar unfortunately. All issues are still under investigation by support and developement, so I understand. thanks ... View more

Looking back: v9.0.4.x no issues noticed v9.0.5.x noticed KVStore issues  v9.1.0.1 noticed all sort of issue including not working 'sendemail'  (upgraded as I had hoped/assumed to be fixed) v9.1.0.2 still the same issues + new strange behaviour : only reading 3 columns from any csv file (upgraded as I hoped/assumed to be fixed) Nb. Have a zoom meeting with Splunk support later this afternoon thanks Reminder: running on Windows 2019 standard,  Xeon CPU, 16 cores, 64Gb , 2Tb ssd ... View more

After the upgrade to v9.1.0.2, last weekend - hoping and assuming some issues were adressed and fixed, as noted in this post, now another issue arised, which make our production-instance even more unreliable. When using an siimple '| inputlookup email.csv' search it only show 3 columns !  On our fresh build test-server with instance v91.0.2 the same serahc produce the correct number of columns. Has somebody experinced this same issue or have a glue why or have a tip. Or what do I miss here? By the way. the sendemail did not work either after upgrade to v91.0.2, after restoring an sendemail.py file from backup v9.0.4 (as I described in this post  earlier sendemail was working again, for now. But what is next! 😞  Nb. Sorry, I really think Splunk should pay more attention to testing and quality control before releasing new versions.  thanks ... View more

Update,  finally I succeed by using the subsearch as was mentioned before + creating 3x6=18 daily scheduled reports. (6 machines)       Dahboard to split in 2 tables panel... Nb. Unfortunately I was not able to make a historical search all-in-one! Thanks ... View more

update: after upgrade to Splunk Enterprise v9.1.0.2 , "sendemail" issue still exist.   ... View more

Update: Last Friday Spiunk Support responded: problem with "sendemail" was identified as a bug and will be solved in next version. Nb. Just noticed version v9.1.0.2 but there was no mention of/refering to this specific fix. I will wait for next version. Sendemail is still working after restoring the senemail.py form backup v9.0.5 ... View more

Update: My main problem was not able to send critical Alerts and other message by email any more after upgradding to v9.1.0.1. Finally I was able to solve this by copying the latest 'sendemail.py" from my backup of v9.0.5. into directory: ....\Splunk\etc\apps\search\bin .  After that I could send email as before. Note: The "sendemail.py" that I copied from backup seems to be 2 lines longer than the new one installed by v9.1.0.1. Did not investigate it further, what the differences is. Busy to correct temp workarounds and made me and my users happy Splunking again. 🙂 Other issues are still open, but for now no showstoppers. Have send this info also to Splunk support but did not here back from them yet.     Thanks ... View more

Thanks, I will do. 😀 ... View more

OK I will try: “splunk cmd OpenSSL” instead of just “openssl”. Thanks ... View more

Hi Both account belongs to me but have different domains. (and no trust-relation in place) Manual sending is no problem and use this daily. It was working for years now based on O365, it sopped after the upgarde. Besides no messages were received form MS that O365 have changed their relay rules. So for now I suspect something not OK in the Splunk 9.1.0.1 environment. Thanks   ... View more

Update: Yesterday we had a remote session (Zoom) with Splunk Support. No quick fix found. It needs to be investigated further: Their observations: ------------------ - After upgrading the Splunk version to 9.1.0.1, the sendemail command is not working it is giving some errors on the production instance and test instance. command="sendemail", The email domains of the recipients are not among those on the allowed domain list. while sending mail to: a.pietersen@....  command="sendemail", (530, b'5.7.57 Client not authenticated to send mail. [AS4P195CA0039.EURP195.PROD.OUTLOOK.COM 2023-07-25T10:35:23.523Z 08DB887EFF8B2458]', 'pietersen@i....) while sending mail to: a.pietersen@.... - The top of the screen showed a loading icon as I was reviewing the Server settings for the Email settings. Nothing can be done on the Email settings page. - Additionally, we attempted to restart the UI by accessing the server controls. The loading icon at the top of the page appeared on the server control page as well, indicating the same problem. - Additionally, the Splunk assist page is not working and displays the error message "Error Loading Splunk Assist." - Even though we attempted to restart the search head using the CLI, the errors and issues persisted after the restart. ------------------ Thanks, hope they can find a remedy soon, because I am out of my option for now.  ... View more

sorry, both on test-server and on production server shows:     ... View more

On my Test-server, I have copied a fresh server.conf from default directory and changed the license manager field. It is coming up and I can login but but still hangs on endless "loading..." when trying to change something (like restart) Also Assist show an error:   ... View more

Hi, looked in server.conf: (both test server + production) Sorry , can not find any path or refence to certificates? test-server:     ... View more

Update: SORRY, AT THE MOMENT NOT MUCH HAPPY SPLUNKING HERE WE RUN A 2nd Splunk Enterprise server (FOR TEST/DEV etc) and most issue mentioned below shows on both machines…. Nb. A week before we upgraded our production machine we installed v9.0.1 on our test-server and we only noticed point 5 and 6 but were not alarmed by the things that would come later after we decided to go forward. It came to our attention after we did not see any Alerts and other emails were send. After upgrade to v9.1.0.1 Splunk Enterprise, (single instance), last weekend (15 Juli 2023) + changing admin password as was suggested by Assist (which throws an error today / now !?) 1) Message when using sendemail: Smpt setting: O365   Checked the login on O365, ofcourse ITEM 1 IS STILL NOT SOLVED 2) Assist stopped running??? ITEM 2 IS STILL NOT SOLVED 3) Also issue with: 3a 3b -after restart:  ITEM  3a and 3b IS STILL NOT SOLVED 4) new GUI / layout ?  ITEM  4 IS NOT ANSWERED YET 5) Annoying and not working “Don’t show this again” message on every page. Just stepping to another dashboard on the same server/domain ??   ITEM 5 HAS BEEN SOLVED 6) endless waiting:        ITEM  6 IS STILL NOT SOLVED – ANY CONFIG CHANGED IN WEB GUI IS NOT WORKING AND ANY CONFIG PAGE KEEPS HANGING. "Loading"         ... View more

Our https certificate (web) is not expired. So what and where should I remove this old cert reference. We only run Indexer, SH , licens Manager and Kvstore process....and nothing else Please note: we run a small and single Splunk Enterprise instance on a Intel 16core , 64Gb mem, and 2Tb hardware under windows2019. BTW How many certs do I need to run and manage in v9.1.0.1?  Thanks ... View more

Hi Romux72 yes we are on v9.1.0.1 I will try: [feature:dashboards_csp] enable_dashboards_redirection_restriction=false result after restart : This seems to work. But will need to test it with different browsers. Sofar, Sogood 🙂 ! Thanks Romux72 ... View more

Also explicit naming of a domain(s) (even its own) doe not seem to work [feature:dashboards_csp] dashboards_trusted_domain.everything=*.<dom1>.nl dashboards_trusted_domain.everything=*.<dom2>.eu dashboards_trusted_domain.everything=*..<dom3>.storage ?? ... View more

Added/tried: in web-feature.conf file: Enable_dashboards_redirection_restriction=false ->  stil redirecting warnings or Enable_dashboards_redirection_restriction=0 ->  stil redirecting warnings .  . What do I miss here?. Document link was not conclusive to me, I want to be able to disable it completely thanks ... View more

mongod.log shows:   2023-07-21T21:14:54.507+0200 W CONTROL [main] Option: sslMode is deprecated. Please use tlsMode instead. 2023-07-21T21:14:54.508+0200 W CONTROL [main] Option: sslCipherConfig is deprecated. Please use tlsCipherConfig instead. 2023-07-21T21:14:54.508+0200 W CONTROL [main] Option: sslAllowConnectionsWithoutCertificates is deprecated. Please use tlsAllowConnectionsWithoutCertificates instead. 2023-07-21T21:14:54.508+0200 W CONTROL [main] Option: sslAllowInvalidHostnames is deprecated. Please use tlsAllowInvalidHostnames instead. 2023-07-21T21:14:54.508+0200 W CONTROL [main] Option: sslAllowInvalidCertificates is deprecated. Please use tlsAllowInvalidCertificates instead. 2023-07-21T21:14:54.508+0200 W CONTROL [main] Option: sslCertificateSelector is deprecated. Please use tlsCertificateSelector instead. 2023-07-21T19:14:54.513Z W CONTROL [main] net.tls.tlsCipherConfig is deprecated. It will be removed in a future release. 2023-07-21T19:14:54.528Z W NETWORK [main] sslCipherConfig parameter is not supported with Windows SChannel and is ignored. 2023-07-21T19:14:54.529Z W NETWORK [main] sslCipherConfig parameter is not supported with Windows SChannel and is ignored. 2023-07-21T19:14:54.529Z F NETWORK [main] The provided SSL certificate is expired or not yet valid. 2023-07-21T19:14:54.529Z F - [main] Fatal Assertion 50755 at src\mongo\util\net\ssl_manager_windows.cpp 1609 2023-07-21T19:14:54.529Z F - [main] \n\n***aborting after fassert() failure\n\n   Where and what can I change to get it working again or ignore these TLS SSL and/expired messages? Where to renew these "indoor"  TLS certs  (self-signed cert I assume and hope) ? Splunk.log shows: 07-22-2023 11:05:55.181 +0200 WARN SearchOperator:kv [13064 SchedulerThread] - Could not find a transform named REPORT-USPS-OFFLINE-CSV 07-22-2023 11:05:55.181 +0200 WARN SearchOperator:kv [13064 SchedulerThread] - Could not find a transform named REPORT-Camera1 07-22-2023 11:05:55.181 +0200 WARN SearchOperator:kv [13064 SchedulerThread] - Could not find a transform named REPORT-usps-off4 07-22-2023 11:05:55.181 +0200 WARN SearchOperator:kv [13064 SchedulerThread] - Could not find a transform named REPORT-USPS-OFFLINE-CSV 07-22-2023 11:05:55.181 +0200 WARN SearchOperator:kv [13064 SchedulerThread] - Could not find a transform named REPORT-usps-off4 07-22-2023 11:05:55.181 +0200 WARN SearchOperator:kv [13064 SchedulerThread] - Could not find a transform named REPORT-USPS-OFFLINE-CSV 07-22-2023 11:05:55.181 +0200 WARN SearchOperator:kv [13064 SchedulerThread] - Could not find a transform named REPORT-Camera1 07-22-2023 11:05:55.213 +0200 INFO NoahSearchPeerFetcher [13064 SchedulerThread] - Fetch requested. sid=scheduler__nobody__search__RMD5883a9bd5121d9759_at_1690016700_59 use_cache=1 07-22-2023 11:06:00.268 +0200 INFO ExecProcessor [10852 ExecProcessor] - setting reschedule_ms=59732, for command="D:\Program Files\Splunk\bin\Python3.exe" "D:\Program Files\Splunk\etc\apps\search\bin\quarantine_files.py" Thanks ... View more

Hi isoutamo I ran the kwstore diag, as admin:  not sure if this is TLS related? also do not understand why I should have TLS enabled to acces to something running on the same single server/Splunk Enterprise instance? where can I disable this in conf file? or how to renwe cert?? thanks   ... View more

Thanks I will check...   ... View more

Hi, As far as I experience with Splunk Enterprise V9.1.0 this is still not solved/available. Please let me know how to disable this "feature" 😞 Ashleyp ... View more

Hi  Elizabeth, We justed upgraded to v9.1 Splunk Enterprise. Can you tell me where I can disable this "feature" AshleyP ... View more