topic Re: Windows logs collection with WMI arrive with delay? in Getting Data In

Windows logs collection with WMI arrive with delay?

SplunkExplorer — Wed, 28 Jun 2023 18:25:03 GMT

Hi Splunkers,

for our customer we collect log from Windows systems. The main configuration details are:

Logs go from DCs to a dedicated HF and then to Splunk Cloud, so the flow is: DCs -> HF -> Splunk Cloud
Due customer policy, we avoided UF and used the WMI Collection, so on HF we configured, as Data Input, the Remote event log Collection.
Configuring Remote event log Collection, we put one DC hostname in box Collect logs from this host and then we added the remaining ones in the box to add additional hosts. I mean: with only one Remote event Collection data inputs, we are collection logs from alla DCs, and they are 12.
We collect following data type:
- Application
- System
- Security
- DNS
- PoweShell
Currently, we applied no blacklist and/or other filter mechanis, so we are collecting all logs from above category
Our HF has the recommened system requirements.

Yesterday we completed this configuration and started to collect logs. The issue we are facing is that logs arrive with a delay, which is always around 30-60 minutes. So, we have to understand why.

Our suspect is that we have not a single root cause, but a set and it is:

Use of WMI insted of forwarder, that could be problematic if we have multiple hosts, as stated in this Splunk Community Thread
Collection of all logs without filtering anything; I mean that for above categories we collect all related EventCode occurrences.
A "burst" in sending logs, cause we started collection from all DCs in the same time
Configured only one Remote event log Collection, but we think this have a minimum weight on performance issues.

Based on this, if all our assumptions are correct, considering that customer for sure will not enable UF installation, we thougth to:

Excluding unwanted logs with inputs.conf in HF
Evaluate if create separate Remote event log Collection input, in worst case one for every DC.

Do you think guys this is fine? Our main doubt currently is: have we detected all issue causes? Are our fixes the right ones?

Re: Windows logs collection with WMI arrive with delay?

isoutamo — Tue, 04 Jul 2023 20:37:13 GMT

what I have learn/heard is that use WMI only if you haven't any other options and you must collect those logs! It's much better to use UF on those nodes!

As you have 12 DC nodes you probably have quite high event count (1-X kilo events/s)? Definitely you should try to filter and collect only needed event not all from those logs.

r. Ismo

Re: Windows logs collection with WMI arrive with delay?

inventsekar — Wed, 05 Jul 2023 01:43:21 GMT

30-60 minutes ,...is a too much delay. It should not take so long. even 3mins delay itself is unacceptable.

on the Splunk Cloud side, there should not be much of a delay. The delay is only should be on your systems/network side.

pls troubleshoot the networks/connections, etc and update us how it goes..

Re: Windows logs collection with WMI arrive with delay?

isoutamo — Wed, 05 Jul 2023 05:53:40 GMT

If this is a very busy DC there could be hundreds/thousands events per second. With this amount of event and badly designed collection it cannot work. I have seen same kind of situations couple of time where they couldn’t found any suitable solutions to collect all needed events.