This isn't so much a question as a comment. I found that time config to be incorrect.  My logs start like this: {"Time": "29 Jan 2025 03:16:30, PST", The default timestring is expecting a 2 digit year.   %d %b %y %H:%M:%S, %Z   Prior to the update, Splunk was stil able to figure out the time but issed the timezone parameter. In other words, if your heavy forwarder has the same timezone as your zScaler logs you would probably be fine.     ... View more

Is anyone else running into boot-start/permissions issues with the 9.0.0 UF running on Linux using init.d scripts for bootstart? Warning: Attempting to revert the SPLUNK_HOME ownership Warning: Executing "chown -R splunk /opt/splunkforwarder" I am also finding that "./splunk disable boot-start" does not correctly remove the /etc/init.d/splunk script and, contrary to documentation, splunk UF 9.0.0 uses systemd as default. https://docs.splunk.com/Documentation/Splunk/9.0.0/Admin/ConfigureSplunktostartatboottime Also systemd scripts seem to fail getting the permissions needed even when trying to enable-boot as root. A key error I am seeing is "Failed to create the unit file" when running the install. But it seems to be a total fail.     ## When upgrading (from 8.2.5) runuser -l splunk -c "/opt/splunkforwarder/bin/splunk stop" tar -xzvf /tmp/splunkforwarder-9.0.0-6818ac46f2ec-Linux-x86_64.tgz -C /opt chown -R splunk:splunk /opt/splunkforwarder/ runuser -l splunk -c "/opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --no-prompt" runuser -l splunk -c "/opt/splunkforwarder/bin/splunk status" Warning: Attempting to revert the SPLUNK_HOME ownership Warning: Executing "chown -R splunk /opt/splunkforwarder" (NOTE: Seems to be non-impacting)   ### When doing a new install tar -xzvf /tmp/splunkforwarder-9.0.0-6818ac46f2ec-Linux-x86_64.tgz -C /opt chown -R splunk:splunk /opt/splunkforwarder [root]# sudo -H -u splunk /opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --no-prompt Warning: Attempting to revert the SPLUNK_HOME ownership Warning: Executing "chown -R splunk /opt/splunkforwarder" This appears to be your first time running this version of Splunk. IMPORTANT: Because an admin password was not provided, the admin user will not be created. You will have to set up an admin username/password later using user-seed.conf. Creating unit file... Current splunk is running as non root, which cannot operate systemd unit files. Please create it manually by 'sudo splunk enable boot-start' later. Failed to create the unit file. Please do it manually later. Splunk> Now with more code! sudo -H -u splunk /opt/splunkforwarder/bin/splunk status Warning: Attempting to revert the SPLUNK_HOME ownership Warning: Executing "chown -R splunk /opt/splunkforwarder" splunkd is running (PID: 3132350). splunk helpers are running (PIDs: 3132354).   # sudo -H -u splunk /opt/splunkforwarder/bin/splunk stop Warning: Attempting to revert the SPLUNK_HOME ownership Warning: Executing "chown -R splunk /opt/splunkforwarder" Stopping splunkd... Shutting down. Please wait, as this may take a few minutes. . [ OK ] Stopping splunk helpers... [ OK ] Done. # /opt/splunkforwarder/bin/splunk enable boot-start -user splunk Systemd unit file installed by user at /etc/systemd/system/SplunkForwarder.service. Configured as systemd managed service. systemctl start SplunkForwarder.service Job for SplunkForwarder.service failed because the control process exited with error code. See "systemctl status SplunkForwarder.service" and "journalctl -xe" for details. systemctl status SplunkForwarder.service ● SplunkForwarder.service - Systemd service file for Splunk, generated by 'splunk enable boot-start' Loaded: loaded (/etc/systemd/system/SplunkForwarder.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Tue 2022-06-21 12:58:55 UTC; 27s ago Process: 3141480 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/SplunkForwarder.service (code=exited, status=0/SUCCES> Process: 3141478 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/SplunkForwarder.service (code=exited, status=0/SUCCESS) Process: 3141477 ExecStart=/opt/splunkforwarder/bin/splunk _internal_launch_under_systemd (code=exited, status=203/EXEC) Process: 3141475 ExecStartPre=/bin/bash -c chown -R splunk:splunk /opt/splunkforwarder (code=exited, status=0/SUCCESS) Main PID: 3141477 (code=exited, status=203/EXEC) Jun 21 12:58:55 <host> systemd[1]: SplunkForwarder.service: Failed with result 'exit-code'. Jun 21 12:58:55 <host> systemd[1]: Failed to start Systemd service file for Splunk, generated by 'splunk enable boot-start'. Jun 21 12:58:55 <host> systemd[1]: SplunkForwarder.service: Service RestartSec=100ms expired, scheduling restart. Jun 21 12:58:55 <host> systemd[1]: SplunkForwarder.service: Scheduled restart job, restart counter is at 5. Jun 21 12:58:55 <host> systemd[1]: Stopped Systemd service file for Splunk, generated by 'splunk enable boot-start'. Jun 21 12:58:55 <host> systemd[1]: SplunkForwarder.service: Start request repeated too quickly. Jun 21 12:58:55 <host> systemd[1]: SplunkForwarder.service: Failed with result 'exit-code'. Jun 21 12:58:55 <host> systemd[1]: Failed to start Systemd service file for Splunk, generated by 'splunk enable boot-start'. ... View more