Why do I get an error when doing an offline SC4S installation

[ASGrover]

Hi there, 

I am trying to do an air-gapped offline container install of SC4S on RHEL 9 with Podman. And I keep getting the following errors. I've provided a list of steps of what I have done so far and the link to the documentation that I have followed and files that I have needed to edit. Any and all suggestions will be massively appreciated! 

Installing a container while offline: https://splunk.github.io/splunk-connect-for-syslog/main/gettingstarted/docker-podman-offline/

I followed step 1 - 5 perfectly fine.

Below is systemd unit file /lib/systemd/system/sc4s.service that I am currently using: 

[Unit]
Description=SC4S Container
Wants=NetworkManager.service network-online.target
After=NetworkManager.service network-online.target

[Install]
WantedBy=multi-user.target

[Service]
Environment="SC4S_IMAGE=sc4slocal:latest"

# Required mount point for syslog-ng persist data (including disk buffer)
Environment="SC4S_PERSIST_MOUNT=splunk-sc4s-var:/var/lib/syslog-ng"

# Optional mount point for local overrides and configurations; see notes in docs
Environment="SC4S_LOCAL_MOUNT=/opt/sc4s/local:/etc/syslog-ng/conf.d/local:z"

# Optional mount point for local disk archive (EWMM output) files
Environment="SC4S_ARCHIVE_MOUNT=/opt/sc4s/archive:/var/lib/syslog-ng/archive:z"

# Map location of TLS custom TLS
Environment="SC4S_TLS_MOUNT=/opt/sc4s/tls:/etc/syslog-ng/tls:z"

TimeoutStartSec=0

#ExecStartPre=/usr/bin/podman pull $SC4S_IMAGE

# Note: /usr/bin/bash will not be valid path for all OS
# when startup fails on running bash check if the path is correct
ExecStartPre=/usr/bin/bash -c "/usr/bin/systemctl set-environment SC4SHOST=$(hostname -s)"

# Note: Prevent the error 'The container name "/SC4S" is already in use by container <container_id>. You have to remove (or rename) that container to be able to reuse that name.'
ExecStartPre=/usr/bin/bash -c "/usr/bin/podman rm SC4S > /dev/null 2>&1 || true"
ExecStart=/usr/bin/podman run \
        -e "SC4S_CONTAINER_HOST=${SC4SHOST}" \
        -v "$SC4S_PERSIST_MOUNT" \
        -v "$SC4S_LOCAL_MOUNT" \
        -v "$SC4S_ARCHIVE_MOUNT" \
        -v "$SC4S_TLS_MOUNT" \
        --env-file=/opt/sc4s/env_file \
        --health-cmd="/usr/sbin/syslog-ng-ctl healthcheck --timeout 5" \
        --health-interval=2m --health-retries=6 --health-timeout=5s \
        --network host \
        --name SC4S \
        --rm $SC4S_IMAGE

Restart=on-failure

Once done, I then create a local volume, using the command shown below: 

sudo podman volume create splunk-sc4s-var

I also created the following subdirectories: 

• /opt/sc4s/local
• /opt/sc4s/archive
• /opt/sc4s/tls 

And I also created the env_file aswell, this is shown below:

SC4S_DEST_SPLUNK_HEC_DEFAULT_URL=https://indexer1:8088,https://indexer2:8088,https://indexer3:8088
SC4S_DEST_SPLUNK_HEC_DEFAULT_TOKEN=<HEC Splunk Management Token Here>
#Uncomment the following line if using untrusted SSL certificates
#SC4S_DEST_SPLUNK_HEC_DEFAULT_TLS_VERIFY=no

I then do the following:

• sudo systemctl daemon-reload
• sudo systemctl enable sc4s
• sudo systemctl start sc4s
• podman logs SC4S

Here, I see the following events: 

curl failed to verify the legitimacy of the server and therefore could not establish a sure connection to it. To learn more about this situation and how to fix it, please visit the webpage mentioned above. 
SC4S_ENV_CHECK_HEC: Invalid Splunk HEC URL, invalid token, or other HEC connectivity issue index=main, sourcetype=sc4s:fallback
Startup will continue to prevent data loss if this is a transient failure. 

syslog-ng checking config
sc4s version=3.38.1
Configuring the health check port to: 8080
[<timestamp>] [124] [INFO] Starting guicorn 23.0.0
[<timestamp>] [124] [INFO] Listening at: http://0.0.0.0:8000 (124)
[<timestamp>] [124] [INFO] Using worker: sync
[<timestamp>] [124] [INFO] Booting worker with pid: 126
starting syslog-ng
Traceback (most recnet call last):
File "/etc/syslog-ng/syslog-ng.conf{python-global-code:1}", line 13, in splunk_hec_metric_multi_v2"
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe1 in position 505: invalid continuation byte

The Traceback and the UnicodeDecodeError repeat itself constantly 

[ASGrover]

Just to add onto this, this is being added to an distributed clustered environment. Hence why the splunk management is the HEC_DEFAULT_TOKEN, and the Indexers 1-3 are the provided IP addresses for the HEC_DEFAULT_URL