Hi Splunkers,
for our customer we collect log from Windows systems. The main configuration details are:
- Logs go from DCs to a dedicated HF and then to Splunk Cloud, so the flow is: DCs -> HF -> Splunk Cloud
- Due customer policy, we avoided UF and used the WMI Collection, so on HF we configured, as Data Input, the Remote event log Collection.
- Configuring Remote event log Collection, we put one DC hostname in box Collect logs from this host and then we added the remaining ones in the box to add additional hosts. I mean: with only one Remote event Collection data inputs, we are collection logs from alla DCs, and they are 12.
- We collect following data type:
- Application
- System
- Security
- DNS
- PoweShell
- Currently, we applied no blacklist and/or other filter mechanis, so we are collecting all logs from above category
- Our HF has the recommened system requirements.
Yesterday we completed this configuration and started to collect logs. The issue we are facing is that logs arrive with a delay, which is always around 30-60 minutes. So, we have to understand why.
Our suspect is that we have not a single root cause, but a set and it is:
- Use of WMI insted of forwarder, that could be problematic if we have multiple hosts, as stated in this Splunk Community Thread
- Collection of all logs without filtering anything; I mean that for above categories we collect all related EventCode occurrences.
- A "burst" in sending logs, cause we started collection from all DCs in the same time
- Configured only one Remote event log Collection, but we think this have a minimum weight on performance issues.
Based on this, if all our assumptions are correct, considering that customer for sure will not enable UF installation, we thougth to:
- Excluding unwanted logs with inputs.conf in HF
- Evaluate if create separate Remote event log Collection input, in worst case one for every DC.
Do you think guys this is fine? Our main doubt currently is: have we detected all issue causes? Are our fixes the right ones?
