Getting Data In

Windows logs collection with WMI arrive with delay?

SplunkExplorer
Communicator

Hi Splunkers,

for our customer we collect log from Windows systems. The main configuration details are:

  • Logs go from DCs to a dedicated HF and then to Splunk Cloud, so the flow is: DCs -> HF -> Splunk Cloud
  • Due customer policy, we avoided UF and used the WMI Collection, so on HF we configured, as Data Input, the Remote event log Collection.
  • Configuring Remote event log Collection, we put one DC hostname in box Collect logs from this host and then we added the remaining ones in the box to add additional hosts. I mean: with only one Remote event Collection data inputs, we are collection logs from alla DCs, and they are 12.
  • We collect following data type:
    • Application
    • System
    • Security
    • DNS
    • PoweShell
  • Currently, we applied no blacklist and/or other filter mechanis, so we are collecting all logs from above category
  • Our HF has the recommened system requirements.

Yesterday we completed this configuration and started to collect logs. The issue we are facing is that logs arrive with a delay, which is always around 30-60 minutes. So, we have to understand why.

Our suspect is that we have not a single root cause, but a set and it is:

  • Use of WMI insted of forwarder, that could be problematic if we have multiple hosts, as stated in this Splunk Community Thread 
  • Collection of all logs without filtering anything; I mean that for above categories we collect all related EventCode occurrences.
  • A "burst" in sending logs, cause we started collection from all DCs in the same time
  • Configured only one Remote event log Collection, but we think this have a minimum weight on performance issues.

Based on this, if all our assumptions are correct, considering that customer for sure will not enable UF installation, we thougth to:

  • Excluding unwanted logs with inputs.conf in HF
  • Evaluate if create separate Remote event log Collection input, in worst case one for every DC.

Do you think guys this is fine? Our main doubt currently is: have we detected all issue causes? Are our fixes the right ones?

Labels (4)
0 Karma
1 Solution

isoutamo
SplunkTrust
SplunkTrust

Hi

what I have learn/heard is that use WMI only if you haven't any other options and you must collect those logs! It's much better to use UF on those nodes!

As you have 12 DC nodes you probably have quite high event count (1-X kilo events/s)? Definitely you should try to filter and collect only needed event not all from those logs.

r. Ismo

View solution in original post

inventsekar
Ultra Champion

30-60 minutes ,...is a too much delay. It should not take so long. even 3mins delay itself is unacceptable.

on the Splunk Cloud side, there should not be much of a delay. The delay is only should be on your systems/network side. 

pls troubleshoot the networks/connections, etc and update us how it goes.. 

0 Karma

isoutamo
SplunkTrust
SplunkTrust
If this is a very busy DC there could be hundreds/thousands events per second. With this amount of event and badly designed collection it cannot work. I have seen same kind of situations couple of time where they couldn’t found any suitable solutions to collect all needed events.
0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

what I have learn/heard is that use WMI only if you haven't any other options and you must collect those logs! It's much better to use UF on those nodes!

As you have 12 DC nodes you probably have quite high event count (1-X kilo events/s)? Definitely you should try to filter and collect only needed event not all from those logs.

r. Ismo

Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...