Hi there,  I am trying to do an air-gapped offline container install of SC4S on RHEL 9 with Podman. And I keep getting the following errors. I've provided a list of steps of what I have done so far and the link to the documentation that I have followed and files that I have needed to edit. Any and all suggestions will be massively appreciated!  Installing a container while offline: https://splunk.github.io/splunk-connect-for-syslog/main/gettingstarted/docker-podman-offline/ I followed step 1 - 5 perfectly fine. Below is systemd unit file /lib/systemd/system/sc4s.service that I am currently using:  [Unit] Description=SC4S Container Wants=NetworkManager.service network-online.target After=NetworkManager.service network-online.target [Install] WantedBy=multi-user.target [Service] Environment="SC4S_IMAGE=sc4slocal:latest" # Required mount point for syslog-ng persist data (including disk buffer) Environment="SC4S_PERSIST_MOUNT=splunk-sc4s-var:/var/lib/syslog-ng" # Optional mount point for local overrides and configurations; see notes in docs Environment="SC4S_LOCAL_MOUNT=/opt/sc4s/local:/etc/syslog-ng/conf.d/local:z" # Optional mount point for local disk archive (EWMM output) files Environment="SC4S_ARCHIVE_MOUNT=/opt/sc4s/archive:/var/lib/syslog-ng/archive:z" # Map location of TLS custom TLS Environment="SC4S_TLS_MOUNT=/opt/sc4s/tls:/etc/syslog-ng/tls:z" TimeoutStartSec=0 #ExecStartPre=/usr/bin/podman pull $SC4S_IMAGE # Note: /usr/bin/bash will not be valid path for all OS # when startup fails on running bash check if the path is correct ExecStartPre=/usr/bin/bash -c "/usr/bin/systemctl set-environment SC4SHOST=$(hostname -s)" # Note: Prevent the error 'The container name "/SC4S" is already in use by container <container_id>. You have to remove (or rename) that container to be able to reuse that name.' ExecStartPre=/usr/bin/bash -c "/usr/bin/podman rm SC4S > /dev/null 2>&1 || true" ExecStart=/usr/bin/podman run \ -e "SC4S_CONTAINER_HOST=${SC4SHOST}" \ -v "$SC4S_PERSIST_MOUNT" \ -v "$SC4S_LOCAL_MOUNT" \ -v "$SC4S_ARCHIVE_MOUNT" \ -v "$SC4S_TLS_MOUNT" \ --env-file=/opt/sc4s/env_file \ --health-cmd="/usr/sbin/syslog-ng-ctl healthcheck --timeout 5" \ --health-interval=2m --health-retries=6 --health-timeout=5s \ --network host \ --name SC4S \ --rm $SC4S_IMAGE Restart=on-failure Once done, I then create a local volume, using the command shown below:  sudo podman volume create splunk-sc4s-var I also created the following subdirectories:  /opt/sc4s/local /opt/sc4s/archive /opt/sc4s/tls  And I also created the env_file aswell, this is shown below: SC4S_DEST_SPLUNK_HEC_DEFAULT_URL=https://indexer1:8088,https://indexer2:8088,https://indexer3:8088 SC4S_DEST_SPLUNK_HEC_DEFAULT_TOKEN=<HEC Splunk Management Token Here> #Uncomment the following line if using untrusted SSL certificates #SC4S_DEST_SPLUNK_HEC_DEFAULT_TLS_VERIFY=no   I then do the following: sudo systemctl daemon-reload sudo systemctl enable sc4s sudo systemctl start sc4s podman logs SC4S Here, I see the following events:  curl failed to verify the legitimacy of the server and therefore could not establish a sure connection to it. To learn more about this situation and how to fix it, please visit the webpage mentioned above.  SC4S_ENV_CHECK_HEC: Invalid Splunk HEC URL, invalid token, or other HEC connectivity issue index=main, sourcetype=sc4s:fallback Startup will continue to prevent data loss if this is a transient failure.  syslog-ng checking config sc4s version=3.38.1 Configuring the health check port to: 8080 [<timestamp>] [124] [INFO] Starting guicorn 23.0.0 [<timestamp>] [124] [INFO] Listening at: http://0.0.0.0:8000 (124) [<timestamp>] [124] [INFO] Using worker: sync [<timestamp>] [124] [INFO] Booting worker with pid: 126 starting syslog-ng Traceback (most recnet call last): File "/etc/syslog-ng/syslog-ng.conf{python-global-code:1}", line 13, in splunk_hec_metric_multi_v2" UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe1 in position 505: invalid continuation byte The Traceback and the UnicodeDecodeError repeat itself constantly  ... View more

I installed a new license on my Splunk system v.9.0.4 on Windows, where I navigated to the licenses section in the settings, and uploaded a new license before removing the old expired one I had previously. I was then prompted to restart the Splunk environment that to let the new Splunk license take affect. Only for me to be pushed back to the login page, where I can no longer login with both my normal account and the admin account.  Is there anyone who can help. As I can't access my Splunk system anymore.    Thanks ... View more

Hi everyone, I’m currently working with a Splunk distributed clustered environment (v9.4.1), with 3 indexers, 3 search heads and 1 cluster master, on RHEL.  I recently added a second 500GB disk to each indexer in order to separate hot/warm and cold bucket storage. I have set up and mounted the 500GB disks hoping that should differentiate between the /indexes and the /coldstore.  I also edited the indexes.conf file on the cluster master, an example is shown below: [bmc] homePath = /indexes/bmc/db coldPath = /coldstore/bmc/colddb thawedPath = $SPLUNK_DB/bmc/thaweddb repFactor = auto maxDataSize = auto_high_volume I then applied the cluster-bundle as well as gave it a rolling-restart just in case.  Even though (I think) that I have configured everything correctly, when I navigate to the cluster master GUI and go to the path  Settings → Indexer Clustering → Indexes The indexes tab is empty, with none of the default indexes or the custom indexes that I had made. Has anyone encountered this behaviour where indexes do not appear in the Clustering UI, despite valid indexes.conf and bundle deployment? ... View more