What Log Files Should Be Monitored in SUSE Linux for Splunk ES, and How Do They Differ from Standard Linux Monitoring?

[sumanssa]

Hi Splunk Community,

We’re currently onboarding SUSE Linux (SLES/OpenSUSE) logs into Splunk Enterprise Security (ES) and would appreciate some input.

Specifically, I’m looking to understand:

• What log files are most relevant for SUSE Linux when it comes to security-focused use cases in Splunk ES (e.g., authentication, audit, change tracking, endpoint monitoring)?

• How do SUSE Linux log paths and formats differ from standard Linux distributions like RHEL, CentOS, or Ubuntu?

• Are there any known configurations or tuning steps required (e.g., for /var/log/secure, auditd, or firewall logs) to ensure Splunk ES use cases are fully supported?

If anyone has experience with Splunk ES and SUSE integration, I’d love to hear your recommendations on best practices or common challenges.

Thanks in advance!

[LAME-Creations]

Hi,

Onboarding SUSE Linux (SLES/OpenSUSE) logs into Splunk Enterprise Security (ES) for security-focused use cases is a great initiative, and I’d be happy to share insights on the key log files, differences from other Linux distributions, configuration steps, and best practices. Below, I’ll address each of your questions in detail, drawing from general Splunk practices and specific considerations for SUSE Linux, with some references to community insights where applicable.

1. Most Relevant Log Files for Security-Focused Use Cases in Splunk ES

For security-focused use cases in Splunk ES, such as authentication monitoring, audit tracking, change management, and endpoint monitoring, the following SUSE Linux log files are critical. These logs align with Splunk ES’s data models (e.g., Authentication, Change, Endpoint) and support use cases like detecting unauthorized access, privilege escalation, or system changes.

- /var/log/messages (or /var/log/syslog in some configurations):

  - Purpose: General system log capturing a wide range of system events, including security-related messages like sudo commands, system service activities, and kernel messages.

  - Use Cases: Useful for monitoring system-wide events, detecting anomalies (e.g., unexpected service failures), and correlating with other logs for incident investigation.

  - Splunk ES Mapping: Feeds into the *hange and Endpoint data models for tracking system activities.

- /var/log/secure (or /var/log/auth.log in some SUSE configurations):

  - Purpose: Captures authentication-related events, such as successful/failed logins, SSH access, su/sudo usage, and PAM (Pluggable Authentication Module) events.

  - Use Cases: Essential for the Authentication data model in Splunk ES to detect brute-force attacks, unauthorized login attempts, or privilege escalation.

  - Note: On SUSE, the log file is typically /var/log/secure, but verify if your system uses /var/log/auth.log (more common in Debian-based systems like Ubuntu).

- /var/log/audit/audit.log:

  - Purpose: Generated by the auditd daemon, this log records detailed system auditing events, including file access, user management (e.g., changes to /etc/passwd), system calls, and security policy violations.

  - Use Cases: Critical for the Change and Endpoint data models, enabling tracking of file modifications, user account changes, and system call monitoring for compliance (e.g., PCI DSS, CIS benchmarks).

  - Note: Auditd must be properly configured to log meaningful events without overwhelming Splunk with noise (more on tuning below).

- /var/log/firewalld (or firewall-related logs):

  - Purpose: Logs firewall activities, such as blocked connections, allowed traffic, or rule changes, typically managed by firewalld or SuSEfirewall2 (legacy in older SLES versions).

  - Use Cases: Supports the *Network Traffic* and *Intrusion Detection* data models in Splunk ES for monitoring network security events, such as blocked malicious IPs or unauthorized access attempts.

  - Note: Ensure firewalld is enabled and logging is configured (e.g., via LogDenied settings).

- /var/log/apparmor/ (e.g., /var/log/apparmor/audit.log):

  - Purpose: Logs AppArmor events, including profile violations or permitted actions, which are critical for mandatory access control (MAC) monitoring.

  - Use Cases: Useful for detecting attempts to access restricted files or execute unauthorized processes, feeding into the Endpoint data model.

  - Note: AppArmor is commonly used in SUSE for security hardening, so enabling its logging is valuable.

- Application-Specific Logs (e.g Booking.com, Apache, etc.):

  - Purpose: Logs from applications like web servers (/var/log/httpd/ or /var/log/apache2/), databases, or other services running on SUSE systems.

  - Use Cases: Monitor for application-level security events, such as web attacks or unauthorized API access, feeding into the *Web* or *Endpoint* data models.

  - Note: Identify critical applications on your SUSE systems and include their logs based on your security use cases.

- /var/log/zypper.log:

  - Purpose: Logs package management activities (installations, updates, removals) via the zypper package manager, unique to SUSE.

  - Use Cases: Supports the Change data model for tracking software changes that could indicate unauthorized updates or vulnerabilities.

  - Note: Monitor this for compliance and to detect unexpected package modifications.

Recommendation: Start with /var/log/messages, /var/log/secure, and /var/log/audit/audit.log as the core logs for Splunk ES, as they cover most security use cases. Expand to firewall, AppArmor, and application logs based on your environment’s needs. The Splunk Add-on for Unix and Linux (Splunk TA) is a great starting point to configure these inputs, but customize it to avoid collecting unnecessary data.

[PickleRick]

Not quite.

1. Ingesting /var/log/messages indiscriminately will result in a high level of noise. Also with typical syslog-enabled system you will have multiple formats of events which you won't easily parse.

2. There are many issues with TA_nix so advising it as primary source of data is a bit hasty

[PickleRick]

Suse is not that different from other linux distro's. I'd hazard a guess that ubuntu is more non-standard due to its debian heritage.

But more to the point.

With practically any linux distro it's most important to:

1) Determine what data you need to collect (that in turn depends on your use cases; collecting anything "just in case" often just leads to useless pumping up your license).

2) Verifying what sources can/should provide that data and if they are correctly configured to do so (for example - logging from firewall rules requires explicit configuration; auditd might require creating rules to get logs from certain operations and so on).

3) Checking what logging backend are you using for what data (journald/syslog/files...) and possibly reconfiguring it to provide ways of differentiate different kinds of data from each other (for example - reconfigure your local syslog daemon to write to separate files).

And that's pretty much it.