When upgrading the Universal Forwarder using the .tgz on Mac OS , a pop up appears and states the following: The "DeRez" command requires the command line developer tools. Would you like to install the tools now? If 'cancel' is selected, it appears not to affect anything, but I am unsure why this is happening. This appears to be happening when migrating the configuration when upgrading a Splunk UF version on Mac OS. What is the "DeRez" command and what is not being migrated when this is happening?   Thanks!     -- Migration information is being logged to '/Applications/splunkforwarder/var/log/splunk/migration.log.2023-02-01.10-15-52' -- This appears to be an upgrade of Splunk. --------------------------------------------------------------------------------) Splunk has detected an older version of Splunk installed on this machine. To finish upgrading to the new version, Splunk's installer will automatically update and alter your current configuration files. Deprecated configuration files will be renamed with a .deprecated extension. You can choose to preview the changes that will be made to your configuration files before proceeding with the migration and upgrade: If you want to migrate and upgrade without previewing the changes that will be made to your existing configuration files, choose 'y'. If you want to see what changes will be made before you proceed with the upgrade, choose 'n'. Perform migration and upgrade without previewing configuration changes? [y/n] y Migrating to: VERSION=9.0.2 BUILD=17e00c557dc1 PRODUCT=splunk PLATFORM=Darwin-universal It seems that the Splunk default certificates are being used. If certificate validation is turned on using the default certificates (not-recommended), this may result in loss of communication in mixed-version Splunk environments after upgrade. "/Applications/splunkforwarder/etc/auth/ca.pem": already a renewed Splunk certificate: skipping renewal "/Applications/splunkforwarder/etc/auth/cacert.pem": already a renewed Splunk certificate: skipping renewal [DFS] Performing migration. [DFS] Finished migration. [Peer-apps] Performing migration. [Peer-apps] Finished migration. Init script installed at /Library/LaunchDaemons//com.splunk.plist. Init script is configured to run at boot. Splunk> Another one. Checking prerequisites... Management port has been set disabled; cli support for this configuration is currently incomplete. Invalid key in stanza [webhook] in /Applications/splunkforwarder/etc/system/default/alert_actions.conf, line 229: enable_allowlist (value: false). Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug' Checking conf files for problems... Done Checking default conf files for edits... Validating installed files against hashes from '/Applications/splunkforwarder/splunkforwarder-9.0.2-17e00c557dc1-darwin-universal2-manifest' PYTHONHTTPSVERIFY is set to 0 in splunk-launch.conf disabling certificate validation for the httplib and urllib libraries shipped with the embedded Python interpreter; must be set to "1" for increased security All installed files intact. Done All preliminary checks passed. Starting splunk server daemon (splunkd)... Done     ... View more

When on certain pages, my search head says "Loading" and this never goes away. This also causes weird behaviour and javascript errors when viewing certain Splunk administrative pages, such as App installation or LDAP settings. Looking into the code, I find the following:   <script type="text/javascript"> //<![CDATA[ this.messenger = Splunk.Messenger.System.getInstance(); // a misconfigured hierarchy can often derail the module loading, so the 'Loading' string can get stuck there. $("#loading").hide(); //]]> </script>   It says: "a misconfigured hierarchy can often derail the module loading, so the 'Loading' string can get stuck there." What does this mean? How can I see what is the misconfiguration in the hierarchy? What can I look at to see failed module loading? I'm sure that there is a Splunk app that is causing this but, short of trial and error, I am unsure on what could be causing this. Any help would be very useful! Thanks! ... View more

Recently we've been noticing a lot of searches have been getting connection timeouts when trying to query our indexer cluster. We keep getting the message: 2 errors occurred while the search was executing. Therefore, search results might be incomplete. Hide errors. Error connecting: Connect Timeout Timeout error. Timed out waiting for peer searchpeer01. Search results might be incomplete! If this occurs frequently, receiveTimeout in distsearch.conf might need to be increased. Delving into the search.log, we see that we are getting 502 Bad Gateway from the indexer cluster: 06-28-2021 12:45:14.663 ERROR SearchResultTransaction - Got status 502 from https://10.0.0.43:8089/services/streams/search?sh_sid=scheduler__username_aW52X2NpdF9zbm93X3NlYXJjaA__RMD565f4e7f87d23277d_at_1624880700_38630 06-28-2021 12:45:14.663 ERROR SearchResultParser - HTTP error status message from https://10.0.0.43:8089/services/streams/search?sh_sid=scheduler__username_aW52X2NpdF9zbm93X3NlYXJjaA__RMD565f4e7f87d23277d_at_1624880700_38630: Error connecting: Connect Timeout 06-28-2021 12:45:14.663 WARN SearchResultCollator - Failure received on retry collector. _unresolvedRetries=1 06-28-2021 12:45:14.663 WARN SearchResultParserExecutor - Error connecting: Connect Timeout Timeout error. for collector=searchpeer01 06-28-2021 12:45:14.663 ERROR DispatchThread - sid:scheduler__username_aW52X2NpdF9zbm93X3NlYXJjaA__RMD565f4e7f87d23277d_at_1624880700_38630 Timed out waiting for peer searchpeer01. Search results might be incomplete! If this occurs frequently, receiveTimeout in distsearch.conf might need to be increased. Considering  the receiveTimeout is 600 seconds, I don't think that will change anything. I'm not sure where these 502 errors are coming from or what to do about them? Does anyone have any insight into what may be happening? Running version 8.1.3 on the search head and 7.3.3 on the indexer cluster (though planning to upgrade to 8.1.4 as soon as we are able to).   Thanks! ... View more

Hi all, Previously I've used "search_now" to determine the start time of a late-running scheduled search. This appears not to work anymore. Has this been deprecated? I can still use info_min_time, but it's got slightly different functionality than search_now. Thanks! Alex ... View more