SC4S with Keepalived

[L_Petch]

Hello,

2 questions but the second is more of a keepalived question than it is an SC4S question.

First question is what is the advantages of SC4S vs Nginx? From what I can see it doesn't really matter which you use its more preference, is that correct? My current setup is 2 HF's load balancing data between them and will have SC4S implemented on them for syslog.

Second question: I am planning on using keepalived to LB via a VIP, I planned to just track the Podman process in keepalived how you would for apache etc and increment the priority if the process stopped, this would then make keepalived failover the VIP however, the Podman process doesn't get removed if it stops.

What is the best way to achieve this, guessing Podman has its own solution but I rarely use Podman so no idea.

[livehybrid]

Hi @L_Petch 

SC4S (Splunk Connect for Syslog) is purpose-built for syslog ingestion, offering features like automatic source categorization, syslog protocol handling, and Splunk CIM compatibility. NGINX is a general-purpose reverse proxy and load balancer, not a syslog server, so using NGINX for syslog forwarding requires extra configuration and lacks SC4S's syslog-specific features. SC4S is preferred for syslog ingestion into Splunk due to its built-in parsing, normalization, and Splunk integration.

For keepalived with Podman, simply tracking the Podman process may not reliably detect if the SC4S container is healthy or running. Instead, use a custom health check script in keepalived that verifies the SC4S container is running and listening on the expected syslog port (e.g., using podman ps and ss or nc to check port status). This ensures the VIP only fails over when SC4S is truly unavailable.

# Example keepalived health check script
#!/bin/bash
# Check if SC4S container is running
podman ps --filter "name=sc4s" --filter "status=running" | grep sc4s > /dev/null 2>&1 || exit 1
# Check if syslog port (e.g., 514) is listening
ss -ltn | grep ':514 ' > /dev/null 2>&1 || exit 1
exit 0 

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[L_Petch]

Hi @livehybrid 

Apologies I got side tracked by another project and just coming back to this. I am stuck and cannot get this check script to work. I have even tried a simpler "if else " statement but don't seem to get the desired affect. If I change your script to echo rather than exit I get the echo reply I am expecting if the sc4s container state is started or stopped but as soon as it is using exit statements it always exits 1. Below is what I have done, hoping you can spot my mistake.

----------sc4s_health_check_script.sh----------

Only change is making sc4s upper case

# Example keepalived health check script
#!/bin/bash
# Check if SC4S container is running
podman ps --filter "name=SC4S" --filter "status=running" | grep SC4S > /dev/null 2>&1 || exit 1
# Check if syslog port (e.g., 514) is listening
ss -ltn | grep ':514 ' > /dev/null 2>&1 || exit 1
exit 0

----------Keepalived.conf----------

vrrp_script sc4s_health_check {
script "/etc/keepalived/sc4s_health_check_script.sh"
interval 2
weight -20
}

track_script {
sc4s_health_check
}

This is what I get from keepalived and no matter what I change it always returns 1 
Keepalived_vrrp[11414]: VRRP_Script(sc4s_health_check) succeeded
Keepalived_vrrp[11414]: Script `sc4s_health_check` now returning 1
Keepalived_vrrp[11414]: VRRP_Script(sc4s_health_check) failed (exited with status 1)
keepalived_vrrp[11414]: (VI_1) Changing effective priority from 254 to 234

[Prewin27]

@L_Petch Can you make sure your service name filter is not having case sensitivity issue.

Try,

#!/bin/bash

# Check if the container named 'sc4s' is running
if ! podman ps --filter "name=sc4s" --filter "status=running" | grep sc4s > /dev/null 2>&1; then
    exit 1
fi

# Optionally, check if the service inside the container is healthy (e.g., port 514)
if ! ss -ltn | grep ':514 ' > /dev/null 2>&1; then
    exit 1
fi

exit 0

Regards,
Prewin
Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!

[L_Petch]

Hello,

Yes case sensitivity was an issue. I could only get the command to run in CLI if SC4S was in upper case but if it is Upper or Lower case in the script it always exits 1regardless of running state.