Splunk access to RHEL audit.log

[ewok]

Running Splunk 9.3.5 on RHEL 8.  STIG hardened environment. 

The non-Splunk RHEL instances running a Universal Forwarder have no issue access the audit.log files, apparently by virtue of the statement

AmbientCapabilities=CAP_DAC_READ_SEARCH

located in the /etc/systemd/system/SplunkForwarder.service file.  However, the same is not true on the Splunk instance.  They require read access permissions via a file ACL or something.  Where these options all result in multiple STIG compliance findings.  Which each require write ups a vendor (Splunk) dependencies.  

Question - why?  Why can't Splunk access the audit.log files the same way as the UF?  Or is there some way to do the same sort of thing with AmbientCapabilities for Splunkd.Service?

It is tempting to quit collecting these logs with Splunk itself and install UF on the Splunk instances too.

[ewok]

Too follow up.  Using 

[Service]
   AmbientCapabilities=CAP_DAC_READ_SEARCH

This fails under the following conditions:

• If you have an old `Splunkd.service` file, with a line using =!, like the following:

ExecStart=!/opt/splunk/bin/splunk _internal_launch_under_systemd

If so, you will need to recreate the Splunkd.service file.

• If you utilize the "Data inputs --> Files & directories" monitor method for ingest the /var/log/audit/audit,log files this fails.

This works with a current Splunk version (mine is 9.3.5) created Splunkd.service file and using the Splunk_TA_nix script method of ingest using rlog.sh.

Kuddos to @livehybrid for causing me to review and realize I had an out of date Splunkd.service file

[livehybrid]

Hi @ewok 

The Splunk Enterprise systemd unit (splunkd.service) is not shipped with the same AmbientCapabilities=CAP_DAC_READ_SEARCH line that the Universal Forwarder package adds to its own unit (SplunkForwarder.service). 
That line gives the UF process the Linux capability to bypass discretionary access controls (DAC) and read files such as /var/log/audit/audit.log even when the file is mode 0600 and owned by root.
Enterprise installs simply omit that stanza, so splunkd runs with the default capability set and cannot open the audit log unless you relax the permissions or run Splunk as root (both STIG-failures).

You can alter this behaviour for Splunk Enterprise (Full install) by editing the splunkd.service file:

1. Create the override directory (run as root):

   systemctl edit splunkd.service

2. Add theAmbientCapabilities:

   [Service]
      AmbientCapabilities=CAP_DAC_READ_SEARCH

3. Reload systemd and restart Splunk:

   systemctl daemon-reload
      systemctl restart splunkd.service

After the restart the running splunkd should have the same capability that the UF has and can read /var/log/audit/audit.log without touching file permissions or adding ACLs.

The override should not be overwritten with Splunk package upgrades, but always verify after an upgrade that its still in place.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[PrewinThomas]

@ewok 

UF explicitly supports adding Linux capabilities via the AmbientCapabilities=CAP_DAC_READ_SEARCH setting. But full Splunk Enterprise instance does not utilize AmbientCapabilities, so it lacks these elevated per-file read permissions.

So your options to consider is,

-Use the Universal Forwarder on Splunk server hosts for audit log collection and keep Splunk Enterprise running as non-root.

-Run Splunkd as root (least preferred).

So my recommendation is If strict STIG compliance is essential, the best approach is to install the Universal Forwarder (UF) on your Splunk and use it solely for audit log collection.

#https://help.splunk.com/en/splunk-enterprise/forward-and-process-data/universal-forwarder-manual/9.4...

Regards,
Prewin
Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!

[livehybrid]

Hi @ewok 

The Splunk Enterprise systemd unit (splunkd.service) is not shipped with the same AmbientCapabilities=CAP_DAC_READ_SEARCH line that the Universal Forwarder package adds to its own unit (SplunkForwarder.service).
That line gives the UF process the Linux capability to bypass discretionary access controls (DAC) and read files such as /var/log/audit/audit.log even when the file is mode 0600 and owned by root.
Enterprise installs simply omit that stanza, so splunkd runs with the default capability set and cannot open the audit log unless you relax the permissions or run Splunk as root (both STIG-failures).

You can alter this behaviour for Splunk Enterprise (Full install) by editing the splunkd.service file:

1. Create the override directory (run as root):

   systemctl edit splunkd.service

2. Add theAmbientCapabilities:

   [Service]
      AmbientCapabilities=CAP_DAC_READ_SEARCH

3. Reload systemd and restart Splunk:

   systemctl daemon-reload
      systemctl restart splunkd.service

After the restart the running splunkd should have the same capability that the UF has and can read /var/log/audit/audit.log without touching file permissions or adding ACLs.

The override should not be overwritten with Splunk package upgrades, but always verify after an upgrade that its still in place.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing