Splunk Connect for Syslog (SC4S) suddenly stopped working and cannot start (full story with solution)

[woodcock]

We have a very vanilla SC4S configuration that has been working flawlessly with a cron job to do "service sc4s restart" every night to upgrade.  We just discovered that a few nights ago, it did not come back from this nightly restart.

When examining the journal with this command:
journalctl -b -u sc4s

We see this:
Error response from daemon: pull access denied for splunk/scs, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

This problem could happen to ANYBODY at ANY TIME and it took us a while to complete work around it so I am documenting the whole story here.

[bhargavi]

Hi @woodcock , Thanks for this piece of info. I am facing the same issue right now. As per this documentation 'https://github.com/splunk/splunk-connect-for-syslog/blob/main/docs/gettingstarted/quickstart_guide.m... , we can either use docker or podman to get this started. Since podman was already installed in my linux machine, I chose it. After updating sc4s.service for podman, I am trying to reload deamon and restart sc4s, but syslog service isn't starting. When executing 'journalctl -xe', I am seeing the below error. 

Trying to pull ghcr.io/splunk/splunk-connect-for-syslog/container3:latest...
podman[147844]:Error: initializing source docker://ghcr.io/splunk/splunk-connect-for-syslog/container3:latest: pinging container registry ghcr.io: Get "https://ghcr.io/v2/": dial tcp: lookup ghcr.io on 192.124.60.53:53: no such host

Could you please assist here , am I missing anything?

[woodcock]

I forgot to mention that you also need to comment out the line in the systemd unit file that reaches out to github because the failure of that line causes the whole startup to fail. 

[woodcock · Accepted solution]

The splunk(-company)-wrapped syslog-ng service, "Splunk Connect for Syslog" (AKA SC4S) comes standard with a systemd unit file that reaches out on every startup to github to obtain the latest container image.  This had worked flawlessly since we first setup syslog inputs for the client.  However years later, somebody made a WAF change that blocked connectivity to github, which included our download URL round in the unit file (specifically, ghcr.io/splunk/splunk-connect-for-syslog/container3:latest) and did not properly warn or socialize this fact before doing so.

This caused the sc4s service to be unable to restart because the systemd unit file downloads a fresh image every time before it starts, which it could no longer do.

WARNING, if you setup SC4S the normal way, then you did so as user "root" so you will need to do all of this as user "root" also.

The most immediate solution is to see if there is still an older image around to run by using this command:
docker image ls

You should see something like this:
REPOSITORY                                                                                           TAG         IMAGE ID                 CREATED     SIZE   
ghcr.io/splunk/splunk-connect-for-syslog/container2:2     latest     SomeImageID2     SomeDate    SomeSizeGB

If there is, you can modify the unit file by copying the "IMAGE ID" value (in this case "SomeImageID2") and changing this line:
Environment="SC4S_IMAGE=https://ghcr.io/splunk/splunk-connect-for-syslog/container2:2:latest"
To this:
Environment="SC4S_IMAGE=SomeImageID2"
And also commenting out this line, like this:
#ExecStartPre=/usr/bin/docker pull $SC4S_IMAGE

Then you need to reload systemd like this:
systemctl daemon-reload

This should allow you to start your service immediately as normal:
service sc4s start

Now you have the problem of how do you get the latest image manually (now that the automatic download cannot work) which according to this link:
https://splunk.github.io/splunk-connect-for-syslog/main/upgrade/
is now this:
ghcr.io/splunk/splunk-connect-for-syslog/container3:latest

The following link gave us all of what we need but we had to do it a few times with various options mined from the comments to get it eactly right:
https://stackoverflow.com/questions/37905763/how-do-i-download-docker-images-without-using-the-pull-...

You will first have to install docker someplace that CAN get to the image URL.  If you can run a broswer there, just post the value in your browser and it should redirect to an actual page.  If you only have the CLI there, just use curl to test like this:
curl ghcr.io/splunk/splunk-connect-for-syslog/container3:latest

In our case, we just installed docker on a Windows laptop and then opened powershell to run these 2 commands:
docker pull ghcr.io/splunk/splunk-connect-for-syslog/container3:latest
docker image ls

You should see something like this:
REPOSITORY                                                                                       TAG         IMAGE ID                 CREATED     SIZE   
ghcr.io/splunk/splunk-connect-for-syslog/container3     latest     SomeImageID3     SomeDate    SomeSizeGB

Next you need to export the image to a file like this:
docker save SomeImageID3 --output DockerImageSC4S.tar

Then transfer this to "/tmp" on your SC4S server host however you please and load it like this:
docker load -i /tmp/DockerImageSC4S.tar

Then, of course, you need to re-modify the unit file using the new "SomeImageID3" value instead of "SomeImageID2".