Are there specifics to search to determine if a user is being added to Sudoers through the Splunk UF?

[GarzaREG]

I have a requirement where I have been asked to monitor for new users getting added to Sudoer.  Are there specific activities that we need to search for to determine if a user is being added to sudoers through the Splunk UF?  Was using the following SPL index=nixeventlog usermod type=USER op=add-user-to-shadow-group be sufficient?   

[PickleRick]

Shortening @tscroggins 's answer a bit 😉

Splunk on its own works on the data you give it. So it's not a question of "Splunk search" (and that's the section you posted your question in) but rather a general question how to monitor your infrastructure for the changes to sudoers. You can monitor full contents of sudoers file but that will only tell you when the change was made (more or less) and will not tell you who and how did the change. You can monitor any access to the sudoers file but it can produce false positives. You can monitor some aspects of your general system configuration that your sudoers configuration relies on (like wheel group membership) but that will only give you very specific changes and may not catch non-standard changes.

So it's not splunk question as such, more like  a general unix security question about how to find such thing at all.

[tscroggins]

Hi,

Searching for auditd USER_MGMT audit events is one possible method as you've identified:

index=nixeventlog sourcetype IN (auditd linux:audit) type=USER_MGMT (add-user-to-shadow-group OR add-user-to-group) wheel

You may need to decompose the problem further to detect related activity:

1) Document your sudo implementation. This may include sudo, sssd, etc.

2) Monitor changes to sudo and dependent service implementations, e.g.:

/etc/sudo.conf
/etc/sudoers
/etc/sudoers.d/*
etc.

3) Document known sudo groups, e.g. wheel. If you have multiple group sources, i.e. /etc/group, Active Directory, etc., document those as well.

4) Monitor changes to sudo groups.

CIS Benchmarks, DISA STIGs, and other implementation guides can assist with documentation and implementation based on your operating system.

Linux STIGs, for example, recommend the following audit rules:

-w /etc/sudoers -p wa -k identity
-w /etc/sudoers.d/ -p wa -k identity

I used vi to modify a read-only /etc/sudoers file as root. With Splunk Add-on for Linux* <https://splunkbase.splunk.com/app/3412> installed, auditd correctly configured**, and a linux:audit file (monitor) input enabled***, Splunk indexed the auditd events below.

* You can also use Splunk Add-on for Unix <https://splunkbase.splunk.com/app/833> and Linux and the rlog.sh input to collect auditd data or alternatively, you can use Sysmon for Linux and Splunk Add-on for Sysmon for Linux <https://splunkbase.splunk.com/app/6652>.

** There's a small typo in the add-on documentation. The "best practice" log_format value should be ENRICHED, not ENHANCED. Auditd will fail to start with log_format=ENHANCED. I've sent feedback to the content team.

*** Add the following stanza to $SPLUNK_HOME/etc/apps/Splunk_TA_linux/local. Your Splunk user must have read and execute access to /var/log/audit and read access to /var/log/audit/audit.log*. The /etc/audit/auditd.conf log_group property is one method for granting this access.

# $SPLUNK_HOME/etc/apps/Splunk_TA_linux/local

[monitor:///var/log/audit/audit.log*]
disabled = false
sourcetype = linux:audit

index=main sourcetype=auditd key=identity

type=SYSCALL msg=audit(1681571544.453:1378): arch=c000003e syscall=188 success=yes exit=0 a0=55921780fc90 a1=7f444ef7f1ef a2=5592179c7d10 a3=1c items=1 ppid=2715 pid=245788 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="vi" exe="/usr/bin/vi" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="identity" ARCH=x86_64 SYSCALL=setxattr AUID="splunk" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"
type=SYSCALL msg=audit(1681571544.453:1377): arch=c000003e syscall=91 success=yes exit=0 a0=4 a1=8120 a2=7ffc6b024ab0 a3=1b items=1 ppid=2715 pid=245788 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="vi" exe="/usr/bin/vi" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="identity" ARCH=x86_64 SYSCALL=fchmod AUID="splunk" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"
type=SYSCALL msg=audit(1681571544.452:1376): arch=c000003e syscall=188 success=yes exit=0 a0=55921780fc90 a1=7f444f3ceebe a2=5592179d0be0 a3=1b items=1 ppid=2715 pid=245788 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="vi" exe="/usr/bin/vi" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="identity" ARCH=x86_64 SYSCALL=setxattr AUID="splunk" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"
type=SYSCALL msg=audit(1681571544.450:1375): arch=c000003e syscall=257 success=yes exit=4 a0=ffffff9c a1=55921780fc90 a2=41 a3=1a0 items=2 ppid=2715 pid=245788 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="vi" exe="/usr/bin/vi" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="identity" ARCH=x86_64 SYSCALL=openat AUID="splunk" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"
type=SYSCALL msg=audit(1681571544.449:1374): arch=c000003e syscall=82 success=yes exit=0 a0=55921780fc90 a1=5592179c7e00 a2=fffffffffffffe90 a3=1 items=4 ppid=2715 pid=245788 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="vi" exe="/usr/bin/vi" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="identity" ARCH=x86_64 SYSCALL=rename AUID="splunk" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"

Auditd also logs correlated PATH messages:

index=main sourcetype=linux:audit type=PATH name=/etc/sudoers*

type=PATH msg=audit(1681571544.453:1378): item=0 name="/etc/sudoers" inode=67257464 dev=fd:00 mode=0100440 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:etc_t:s0 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0 OUID="root" OGID="root"
type=PATH msg=audit(1681571544.452:1376): item=0 name="/etc/sudoers" inode=67257464 dev=fd:00 mode=0100640 ouid=0 ogid=0 rdev=00:00 obj=unconfined_u:object_r:etc_t:s0 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0 OUID="root" OGID="root"
type=PATH msg=audit(1681571544.450:1375): item=1 name="/etc/sudoers" inode=67257464 dev=fd:00 mode=0100640 ouid=0 ogid=0 rdev=00:00 obj=unconfined_u:object_r:etc_t:s0 nametype=CREATE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0 OUID="root" OGID="root"
type=PATH msg=audit(1681571544.449:1374): item=3 name="/etc/sudoers~" inode=67765819 dev=fd:00 mode=0100440 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:etc_t:s0 nametype=CREATE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0 OUID="root" OGID="root"
type=PATH msg=audit(1681571544.449:1374): item=2 name="/etc/sudoers" inode=67765819 dev=fd:00 mode=0100440 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:etc_t:s0 nametype=DELETE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0 OUID="root" OGID="root"

Splunk Security Essentials <https://splunkbase.splunk.com/app/3435> includes many examples for monitoring scenarios like the file modification above using Splunk Common Information Model (CIM) <https://splunkbase.splunk.com/app/1621>.

The "Linux Possible Access To Sudoers File" analytic provides an example you can modify to search for possible accesses to */etc/sudoers*:

| tstats `security_content_summariesonly` count min(_time) as firstTime max(_time) as lastTime from datamodel=Endpoint.Processes where Processes.process_name IN("cat", "nano*","vim*", "vi*") AND Processes.process IN("*/etc/sudoers*") by Processes.dest Processes.user Processes.parent_process_name Processes.process_name Processes.process Processes.process_id Processes.parent_process_id 
| `drop_dm_object_name(Processes)` 
| `security_content_ctime(firstTime)` 
| `security_content_ctime(lastTime)` 
| `linux_possible_access_to_sudoers_file_filter`

The "Linux Visudo Utility Execution" analytic provides an example you can use to detect execution of visudo:

| tstats `security_content_summariesonly` count min(_time) as firstTime max(_time) as lastTime from datamodel=Endpoint.Processes where Processes.process_name = visudo by Processes.dest Processes.user Processes.parent_process_name Processes.process_name Processes.process Processes.process_id Processes.parent_process_id Processes.process_guid 
| `drop_dm_object_name(Processes)` 
| `security_content_ctime(firstTime)` 
| `security_content_ctime(lastTime)` 
| `linux_visudo_utility_execution_filter`

The "Linux Sudoers Tmp File Creation" analytic provides an example you can modify to search for changes to *suoders.tmp*, *sudoers~*, *sudoers.swp*, and others:

| tstats `security_content_summariesonly` count min(_time) as firstTime max(_time) as lastTime FROM datamodel=Endpoint.Filesystem where Filesystem.file_path IN ("*sudoers.tmp*") by Filesystem.dest Filesystem.file_name Filesystem.process_guid Filesystem.file_path 
| `drop_dm_object_name(Filesystem)` 
| `security_content_ctime(lastTime)` 
| `security_content_ctime(firstTime)` 
| `linux_sudoers_tmp_file_creation_filter`

Neither Splunk Add-on for Linux nor Splunk Add-on for Unix and Linux support the Endpoint.Filesystem data model out of the box, but we can add baseline support (relative to auditd rules tracking writes and attribute changes) with a few knowledge objects:

# $SPLUNK_HOME/etc/apps/Splunk_TA_linux/local/eventtypes.conf

[linux_audit_endpoint_filesystem]
search = sourcetype=linux:audit (type=PATH)
#tags = endpoint filesystem

# $SPLUNK_HOME/etc/apps/Splunk_TA_linux/local/props.conf

[linux:audit]
EVAL-action = case(type=="PATH" AND nametype=="NORMAL", "modified",\
type=="PATH" AND nametype=="DELETE", "deleted",\
type=="PATH" AND nametype=="CREATE", "created",\
true(), null())
EVAL-file_modify_time = case(type=="PATH", _time, true(), null())
EVAL-process_id = case(type IN ("USER_CMD", "SERVICE_START", "SERVICE_STOP") AND isnotnull(pid), pid, null())
EVAL-user = case(type=="PATH" AND isnotnull(OUID), OUID, true(), null())
EXTRACT-file_path = type=PATH\s+msg=audit\([^)]+\):\s+item=\d+\s+name="(?<file_path>.*/(?<file_name>[^/"]+))

# $SPLUNK_HOME/etc/apps/Splunk_TA_linux/local/tags.conf

[eventtype=linux_audit_endpoint_filesystem]
endpoint = enabled
filesystem = enabled

After refreshing or restarting Splunk, we can use the tstats command to find events where "*sudoers*" was modified, created, or deleted:

| tstats summariesonly=false allow_old_summaries=true fillnull_value=null count min(_time) as firstTime max(_time) as lastTime FROM datamodel=Endpoint.Filesystem where Filesystem.file_path IN ("*sudoers*") by Filesystem.dest Filesystem.file_name Filesystem.file_path

With this background, you can expand audit rules to include files like /etc/passwd, /etc/shadow, /etc/group, and /etc/gshadow and commands like usermod and gpasswd:

-w /etc/passwd -p wa -k identity
-w /etc/group -p wa -k identity
-w /etc/gshadow -p wa -k identity

-a always,exit -F path=/usr/bin/gpasswd -F perm=x -F auid>=1000 -F auid!=unset -k privileged-gpasswd
-a always,exit -F path=/usr/sbin/usermod -F perm=x -F auid>=1000 -F auid!=unset -k privileged-usermod

Searches similar to those above can be used to detect possible changes to group memberships:

| tstats summariesonly=false allow_old_summaries=true fillnull_value=null count min(_time) as firstTime max(_time) as lastTime FROM datamodel=Endpoint.Filesystem where Filesystem.file_path IN ("*/etc/group" "*/etc/gshadow" "*/etc/passwd") by Filesystem.dest Filesystem.file_name Filesystem.file_path