For those of you who are ingesting Sysmon data from workstations -- what's the ingest volume look like for you per day? How many workstations are you collecting from?
We are going to be discussing ingesting this data internally and don't have a starting point yet -- but the plan is to enable sysmon on a handful of workstations and measure the ingest; but we're a few weeks away from that at this point. We've got about 6,000-7,000 workstations on campus we're interested in collecting this data from.
I'm interested in collecting sysmon from workstation and the scope is pretty much equal as you have. Can you, please share any calculation of what is (approximate is good) the volume of symon data per day I should expect from 1K hosts?
This answer highly depends on the XML configuration you apply to Sysmon. With our configuration now, we're seeing an average of 20-25GB/day across ~6,000 endpoints. So you're looking at maybe 4-10GB /day for 1000 hosts. Sometimes our ingest will spike (like during a patch day) and we'll hit 60-70GB a day during peaks.
My suggestion is to start with a minimal configuration instead of enabling everything from the beginning. Add events onto it and be prepared to blacklist or remove a configuration after you've deployed it. Sysmon can be really finicky until you get it to a point where everyone is comfortable with the type of data you're getting vs the amount of licensing being consumed.
Also be aware that depending on the type of events you bring in, you could end up pulling in plaintext passwords from scripts or command line switches into Splunk. You can mask these with a SEDCMD in props.conf on the forwarder side.
The volume of data for the universal forwarder to forward varies significantly depending on your sysmon configuration as well as the activity levels of users and background processes on monitored hosts. For a common logging baseline to plan against, a good starting point would be the IR community maintained SwiftOnSecurity GitHub repo. I would plan for about 160 GB/day for every 1000 hosts sending sysmon data with SwiftOnSecurity config.
If you are expecting to receive such data for thousands of hosts, you will need to carefully consider whether you want to forward the event log entries in XML or Legacy format. The Add-on for Microsoft Sysmon on Splunkbase provides accelerators assuming the data is in XML format. Choosing to forward as XML has some drawbacks in that every field must be extracted from XML at search time, slowing your searches significantly. If you expose the data to users inexperienced in SPL and job inspection, you may want to consider creating a data model up front. If you are not ready for that yet, then I would send the data in legacy format (renderXML = false).
I find the universal forwarders to be highly reliable. The only time(s) I have ever experienced failure in forwarding of data are in cases where (1) our Splunk servers (receiving tier) have been unavailable for extended periods of time due to unplanned outages stemming from human error and (2) when the forwarder outputs events at a sustained rate that exceeds (configurable) maxKBps limits.
I always felt that the UF is too flaky and i have been seeing that breaks quite often and stops forwarding. How is your experience so far? I am looking to do similar thing with atleast 5000 endpoints.