Archive

Mac OS X Sierra - How to get all logs from the Unified Log database?

Couldn't find a similar question to this one. How are people retrieving logs from Mac OS X Sierra that are in the Unified Logging Database? This was a new logging technology released with Sierra (think it's stored in a binary database). It has way better and more detailed logs compared to the deprecated system.log file. There is practically nothing going to the system.log file in newer OS X versions... Ideally, I'd like to output data from the database and append it to the system.log file so it can get picked up with the rest of our old fashioned syslog (and be forwarded by using an old fashioned forwarding server over udp:514.) The asl.conf appears to be superseded by the Unified Logging as well. Any ideas?

Splunk Employee
Splunk Employee

Hi All, There is a Splunk Idea to track this issue: https://ideas.splunk.com/ideas/EID-I-562

You're welcome to go to the idea to follow it, vote for it and to add additional comments. There is active engineering work done on this, the best way to track that progress and help shape the outcome would be to comment in that Splunk Idea.

0 Karma

Communicator

I ended up kludging a pretty generic scripted input that.

Runs the log show command from startdate to enddate.
Greps what you want using an include file.
Greps out stuff with an exclude file.

tar-zipped TA

Communicator

Thanks for posting. How did you managed to deal with "log show" permissions? Is there any other way than putting user "splunk" into the admin group?

dseditgroup -o edit -a splunk -t user admin
0 Karma

Communicator

I don't currently have a mac to test with and I'm not a Mac guy but something like this might work.

Add this to /etc/sudoers to permit the splunk user to run log without a password.
splunk ALL = NOPASSWD: /path/to/log

Edit the ufmacintosh/bin/maclog_monitor.sh and add sudo to the command.

FROM
log show --style syslog --start "$STARTDATE" --end "$ENDDATE" | egrep -f $INCLUDE | egrep -vf $EXCLUDE

TO
sudo /path/to/log show --style syslog --start "$STARTDATE" --end "$ENDDATE" | egrep -f $INCLUDE | egrep -vf $EXCLUDE

Let me know how it goes!

0 Karma

Communicator

Thanks for the quick response and advice. I had to modify config entry a little, but not much.

splunk ALL=(ALL) NOPASSWD: /usr/bin/log

Just the thing I've noticed. In case the "log show" is not allowed to be run or some other exception happens, the script still updates the lastrundate.txt file. I am thinking of modifying the script so it would update the lastrundate.txt file after log show command would be successfully run.

0 Karma

Engager

So are we in a state where basically Splunk does not work with newer OS/X versions, with the new logging system?

Engager

Please provide step-by-step guidance to on how to get logs from Sierra to Splunk

Path Finder

One possibility is to use osquery to pull the data from asl and put it into a file monitored by the splunk forwarder. And of course osquery exposes lots of other stuff you could grab too.

https://medium.com/@clong/osquery-for-security-b66fffdf2daf
https://blog.kolide.com/monitoring-macos-hosts-with-osquery-ba5dcc83122d

This works - the part I'm struggling with is figuring out what to grab.

Working with the log command in Sierra lets you play with the logged data but I don't see any guidance or recommendations on what to grab to meet standard audit requirements. If you can grab everything great - but if you are concerned about license capacity then most of the stuff going to asl looks like noise and should be filtered at the host.

0 Karma

Motivator

https://github.com/droe/xnumon might also help it's "sysmon for macos"

New Member

Bumping xnumon as a pretty complete solution to this problem. You'll need to transform the input to be CIM compliant since there is not an app available at the time but out of the box it's a fairly on par with what sysmon offers.

0 Karma

Explorer

I'd be very happy to add a sample Splunk config for CIM compliant field extraction to xnumon, feel free to submit one on Github in an issue or pull request.

0 Karma

Path Finder

One other rabbit hole I went down to get audit log data is using auditreduce + praudit

Again this works - audit data goes to splunk - but produces mostly noise. It checks a compliance box without being particularly useful.

I'll check out xnumon. Thanks.

0 Karma

Motivator

It is probably best to contact Splunk, if you need the data from unified logging. That way they can push SPL-129734 internally. For now we rely on some scripts from the Unix TA, I have heard that others use https://osquery.io/

0 Karma

Motivator

The Splunkforwarder can be installed and configured to index information from unified logging. There is just no out of the box functionality for that.

0 Karma

New Member

I'm in the same boat. Need to send/ingest the unified logs from 10.12+ clients.

0 Karma

Motivator

For anyone else stumbling accross this question: Splunk has an open enhancement request for this: SPL-129734. If this is something you need opening a case with a reference to this question might accelerate the implementation.

0 Karma

Splunk Employee
Splunk Employee

some docs from macos
https://www.mac4n6.com/blog/2016/11/13/new-macos-sierra-1012-forensic-artifacts-introducing-unified-...

You could create scripted or modular inputs to run the "log show" command and ingest the events.
The difficulty will be :

Engager

@yannK, when do you think Splunk will get built-in support for ingesting logs from OS X Sierra and above?

I was afraid this might be the answer. In our current case we prefer to have real-time logs so that we could use some alerting. Having a script running a tail on the logs output is not ideal, but something I had thought of.

I'm reading your info on modular inputs, but it's a little confusing. I don't see a difference between using that vs a shell script.

0 Karma

Motivator

Hi managed_security what did you end up setting up? Any chance that you could share a modular input if you set on up?

0 Karma