topic Re: Mac OS X Sierra - How to get all logs from the Unified Log database? in Getting Data In

Mac OS X Sierra - How to get all logs from the Unified Log database?

managed_securit — Thu, 15 Jun 2017 15:24:37 GMT

Couldn't find a similar question to this one. How are people retrieving logs from Mac OS X Sierra that are in the Unified Logging Database? This was a new logging technology released with Sierra (think it's stored in a binary database). It has way better and more detailed logs compared to the deprecated system.log file. There is practically nothing going to the system.log file in newer OS X versions... Ideally, I'd like to output data from the database and append it to the system.log file so it can get picked up with the rest of our old fashioned syslog (and be forwarded by using an old fashioned forwarding server over udp:514.) The asl.conf appears to be superseded by the Unified Logging as well. Any ideas?

Re: Mac OS X Sierra - How to get all logs from the Unified Log database?

yannK — Mon, 17 Jul 2017 19:48:13 GMT

some docs from macos
https://www.mac4n6.com/blog/2016/11/13/new-macos-sierra-1012-forensic-artifacts-introducing-unified-logging

You could create scripted or modular inputs to run the "log show" command and ingest the events.
The difficulty will be :

to decide if you want a live tailing, or a backlog collection of the logs.
to specify and maintain a position checkpoint to avoid reindexing the same events over and over. A modular input may be more appropriate. see http://docs.splunk.com/Documentation/Splunk/6.6.0/AdvancedDev/ModInputsIntro

Re: Mac OS X Sierra - How to get all logs from the Unified Log database?

managed_securit — Tue, 18 Jul 2017 14:27:50 GMT

I was afraid this might be the answer. In our current case we prefer to have real-time logs so that we could use some alerting. Having a script running a tail on the logs output is not ideal, but something I had thought of.

I'm reading your info on modular inputs, but it's a little confusing. I don't see a difference between using that vs a shell script.

Re: Mac OS X Sierra - How to get all logs from the Unified Log database?

johanwalles — Thu, 17 Aug 2017 07:52:24 GMT

@yannK, when do you think Splunk will get built-in support for ingesting logs from OS X Sierra and above?

Re: Mac OS X Sierra - How to get all logs from the Unified Log database?

chris — Mon, 02 Oct 2017 12:20:22 GMT

Hi managed_security what did you end up setting up? Any chance that you could share a modular input if you set on up?

Re: Mac OS X Sierra - How to get all logs from the Unified Log database?

chris — Wed, 04 Oct 2017 07:29:18 GMT

For anyone else stumbling accross this question: Splunk has an open enhancement request for this: SPL-129734. If this is something you need opening a case with a reference to this question might accelerate the implementation.

Re: Mac OS X Sierra - How to get all logs from the Unified Log database?

toyboxent — Sat, 14 Oct 2017 02:00:28 GMT

I'm in the same boat. Need to send/ingest the unified logs from 10.12+ clients.

Re: Mac OS X Sierra - How to get all logs from the Unified Log database?

rgoerwit — Thu, 30 Nov 2017 23:29:50 GMT

So are we in a state where basically Splunk does not work with newer OS/X versions, with the new logging system?

Re: Mac OS X Sierra - How to get all logs from the Unified Log database?

chris — Mon, 04 Dec 2017 07:20:03 GMT

The Splunkforwarder can be installed and configured to index information from unified logging. There is just no out of the box functionality for that.

Re: Mac OS X Sierra - How to get all logs from the Unified Log database?

MRSTANG — Wed, 20 Dec 2017 19:39:38 GMT

Please provide step-by-step guidance to on how to get logs from Sierra to Splunk

Re: Mac OS X Sierra - How to get all logs from the Unified Log database?

chris — Thu, 21 Dec 2017 11:57:06 GMT

It is probably best to contact Splunk, if you need the data from unified logging. That way they can push SPL-129734 internally. For now we rely on some scripts from the Unix TA, I have heard that others use https://osquery.io/

Re: Mac OS X Sierra - How to get all logs from the Unified Log database?

bgstein — Fri, 06 Jul 2018 22:48:53 GMT

One possibility is to use osquery to pull the data from asl and put it into a file monitored by the splunk forwarder. And of course osquery exposes lots of other stuff you could grab too.

https://medium.com/@clong/osquery-for-security-b66fffdf2daf
https://blog.kolide.com/monitoring-macos-hosts-with-osquery-ba5dcc83122d

This works - the part I'm struggling with is figuring out what to grab.

Working with the log command in Sierra lets you play with the logged data but I don't see any guidance or recommendations on what to grab to meet standard audit requirements. If you can grab everything great - but if you are concerned about license capacity then most of the stuff going to asl looks like noise and should be filtered at the host.

Re: Mac OS X Sierra - How to get all logs from the Unified Log database?

chris — Mon, 09 Jul 2018 06:02:21 GMT

https://github.com/droe/xnumon might also help it's "sysmon for macos"

Re: Mac OS X Sierra - How to get all logs from the Unified Log database?

dhaynes_ — Thu, 12 Jul 2018 15:28:32 GMT

Bumping xnumon as a pretty complete solution to this problem. You'll need to transform the input to be CIM compliant since there is not an app available at the time but out of the box it's a fairly on par with what sysmon offers.

Re: Mac OS X Sierra - How to get all logs from the Unified Log database?

bgstein — Thu, 12 Jul 2018 17:01:23 GMT

One other rabbit hole I went down to get audit log data is using auditreduce + praudit

Again this works - audit data goes to splunk - but produces mostly noise. It checks a compliance box without being particularly useful.

I'll check out xnumon. Thanks.

Re: Mac OS X Sierra - How to get all logs from the Unified Log database?

droe — Thu, 19 Jul 2018 21:03:43 GMT

I'd be very happy to add a sample Splunk config for CIM compliant field extraction to xnumon, feel free to submit one on Github in an issue or pull request.

Re: Mac OS X Sierra - How to get all logs from the Unified Log database?

dfronck — Wed, 30 Sep 2020 00:06:42 GMT

I ended up kludging a pretty generic scripted input that.

Runs the log show command from start_date to end_date.
Greps what you want using an include file.
Greps out stuff with an exclude file.

tar-zipped TA

Re: Mac OS X Sierra - How to get all logs from the Unified Log database?

MaverickT — Mon, 23 Mar 2020 13:31:36 GMT

Thanks for posting. How did you managed to deal with "log show" permissions? Is there any other way than putting user "splunk" into the admin group?

dseditgroup -o edit -a splunk -t user admin

Re: Mac OS X Sierra - How to get all logs from the Unified Log database?

dfronck — Wed, 30 Sep 2020 04:45:04 GMT

I don't currently have a mac to test with and I'm not a Mac guy but something like this might work.

Add this to /etc/sudoers to permit the splunk user to run log without a password.
splunk ALL = NOPASSWD: /path/to/log

Edit the uf_macintosh/bin/mac_log_monitor.sh and add sudo to the command.

FROM
log show --style syslog --start "$START_DATE" --end "$END_DATE" | egrep -f $INCLUDE | egrep -vf $EXCLUDE

TO
sudo /path/to/log show --style syslog --start "$START_DATE" --end "$END_DATE" | egrep -f $INCLUDE | egrep -vf $EXCLUDE

Let me know how it goes!

Re: Mac OS X Sierra - How to get all logs from the Unified Log database?

MaverickT — Wed, 30 Sep 2020 04:45:07 GMT

Thanks for the quick response and advice. I had to modify config entry a little, but not much.

splunk ALL=(ALL) NOPASSWD: /usr/bin/log

Just the thing I've noticed. In case the "log show" is not allowed to be run or some other exception happens, the script still updates the last_run_date.txt file. I am thinking of modifying the script so it would update the last_run_date.txt file after log show command would be successfully run.