I have been trying to get the Splunk Remote Upgrader for Linux Universal Forwarders 1.0.3 app to work and upgrade our UFs. The UFs are installed on a supported version of Rocky Linux (9) and we used the .tgx file to install and upgrade the UFs to 9.4.7. The UFs are running as the "splunk" user using the command below to set up and we set the splunk user to be able to read our resources okay. splunk enable boot-start -systemd-managed 1 -user splunk Now, when I use the new app on some of my UFs to Splunk 9.4.8/9, etc, the upgrade seems to complete, but Splunk refuses to start up 3 times and then it rolls back the upgrade to 9.4.7. In the history folder, I see that the logs look for the System Unit file (/etc/systemd/system/SplunkForwarder.service) and if it exists, then the Upgrader assumes that the user running Splunk is root:root and then uses chown -R root:root to the Splunk folder and then the splunk user does not have access to the folder and the start fails. The section in the splunk_util.sh where I found this is:         if [ -f "$SPLUNK_CUSTOM_UNIT_FILE_PATH" ] || [ -f "$SPLUNK_DEFAULT_UNIT_FILE_PATH" ]; then             # if any unit file exists, must run splunk as root/sudo             print_info "Found Splunk unit file. Will start Splunk as ROOT/SUDO."             splunk_user="root" It is not true that if the System Unit file (SplunkForwarder.service) exists, then root must be running Splunk - it contains the lines below that tell systemd to start Splunk as the splunk user and the Upgrader should be looking into the file itself to find the real user. User=splunk Group=splunk root is the user who has access to run "systemctl restart SplunkForwarder", but Splunk is not running as root and never has. If I can stop the process and prevent the Upgrader from rolling back to 9.4.7, then I can run a disable boot-start and then do a enable boot-start and it works, but that is a hack, personally?! ... View more

In the Splunk URA, it says that it includes the /etc/apps and /etc/peer-apps folders in the scans, but it does not include the deployment-apps folder as well. Therefore, the process for scanning apps in the deployment-apps folder is to find these in other places within the environment where SplunkWeb is running and then install/update and run there.  There are more and more companies who are using SplunkCloud and the on-prem presence of Splunk is now mostly managed by the Splunk DS, so why can we not have the ability (in the Splunk URA) to scan the deployment-apps folder, so that it makes on-prem upgrades easier? ... View more

Dear All/Splunk Documentation Team, I am trying to get to read the Compatibility between forwarders and Splunk Enterprise indexers document in the Forwarder Manual for 9.0.1 Forwarders, but all I get is the generic "The topic you've asked to see does not apply to the most recent version" when it patently DOES apply: https://docs.splunk.com/Special:SpecialLatestDoc?t=Documentation/Forwarder/latest/Forwarder/Compatibilitybetweenforwardersandindexers but the latest version available is 8.2.5: https://docs.splunk.com/Documentation/Forwarder/8.2.5/Forwarder/Compatibilitybetweenforwardersandindexers Can this document be copied/updated for 8.2.6/7/8 and 9.x, please?   ... View more

Dear All, I have a pretty bare Splunk Universal Forwarder that was installed at 8.2.5 and had no errors on restart, but when I upgraded it to 9.0.0.1, I started to get the following errors? NOTE: These are all in the system/default files (so not my settings): Invalid key in stanza [webhook] in /opt/splunkforwarder/etc/system/default/alert_actions.conf, line 229: enable_allowlist (value: false). Invalid key in stanza [provider:splunk] in /opt/splunkforwarder/etc/system/default/federated.conf, line 20: mode (value: standard). Invalid key in stanza [general] in /opt/splunkforwarder/etc/system/default/federated.conf, line 23: needs_consent (value: true). ... View more

Dear All, I have a Search Head, Deployment Server, Monitoring Console, a Cluster Manager, an Indexer Cluster and two unclustered Indexers. On the Monitoring Console, I get alerts about the IOWaits being high on the two unclustered indexers and this has been happening only since we upgraded to 8.2.5. There is no evidence of any issues, other than this alert in SplunkWeb and I want to disable it. I am using the following KB article: https://docs.splunk.com/Documentation/Splunk/8.2.5/Admin/Healthconf On the Monitoring Console server, I have put the following into the etc\apps\search\local\health.conf file: [feature:iowait] alert:sum_top3_cpu_percs__max_last_3m.disabled = 1 However, I am still getting the appearing in SplunkWeb on the Monitoring Console server. Why is this? Am I configuring the health.conf in the wrong server or the wrong folder, or what? When I run a cmd btool health list, I see the configuration there, but Splunk is not doing as it is being told! If I am doing the wrong thing, even, can someone point me to some documentation that explains what I should be doing? Thanks in advance!  ... View more

I need to upgrade a Search Head Cluster from 7.3.4 to 8.1.9 and I have run the first two commands: splunk upgrade-init shcluster-members splunk edit shcluster-config -manual_detention on We are monitoring the active searches using the following command: splunk list shcluster-member-info | grep "active" And we see: active_historical_search_count:1 active_realtime_search_count:0 And it seemed to never reduce down to 0 for the active_historical_search_count, but after 90 minutes, it seems to have come down to 0. We checked the currently-running searches and found some new searches running on the detention server after 1 hour. We have the following set in the server.conf: decommission_force_finish_idle_time = 0 decommission_node_force_timeout = 300 decommission_search_jobs_wait_secs = 180 ...so why is it taking 90 minutes to stop running savedsearches? We did find some savedsearches that were running for long times and we fixed them, but should not all new searches be moved to another server once it is in manual detention? What can I do to fix this, so that my SHC can be upgraded? ... View more