Getting Data In

Windows logs collection with WMI arrive with delay?

SplunkExplorer
Communicator

Hi Splunkers,

for our customer we collect log from Windows systems. The main configuration details are:

  • Logs go from DCs to a dedicated HF and then to Splunk Cloud, so the flow is: DCs -> HF -> Splunk Cloud
  • Due customer policy, we avoided UF and used the WMI Collection, so on HF we configured, as Data Input, the Remote event log Collection.
  • Configuring Remote event log Collection, we put one DC hostname in box Collect logs from this host and then we added the remaining ones in the box to add additional hosts. I mean: with only one Remote event Collection data inputs, we are collection logs from alla DCs, and they are 12.
  • We collect following data type:
    • Application
    • System
    • Security
    • DNS
    • PoweShell
  • Currently, we applied no blacklist and/or other filter mechanis, so we are collecting all logs from above category
  • Our HF has the recommened system requirements.

Yesterday we completed this configuration and started to collect logs. The issue we are facing is that logs arrive with a delay, which is always around 30-60 minutes. So, we have to understand why.

Our suspect is that we have not a single root cause, but a set and it is:

  • Use of WMI insted of forwarder, that could be problematic if we have multiple hosts, as stated in this Splunk Community Thread 
  • Collection of all logs without filtering anything; I mean that for above categories we collect all related EventCode occurrences.
  • A "burst" in sending logs, cause we started collection from all DCs in the same time
  • Configured only one Remote event log Collection, but we think this have a minimum weight on performance issues.

Based on this, if all our assumptions are correct, considering that customer for sure will not enable UF installation, we thougth to:

  • Excluding unwanted logs with inputs.conf in HF
  • Evaluate if create separate Remote event log Collection input, in worst case one for every DC.

Do you think guys this is fine? Our main doubt currently is: have we detected all issue causes? Are our fixes the right ones?

Labels (4)
0 Karma
1 Solution

isoutamo
SplunkTrust
SplunkTrust

Hi

what I have learn/heard is that use WMI only if you haven't any other options and you must collect those logs! It's much better to use UF on those nodes!

As you have 12 DC nodes you probably have quite high event count (1-X kilo events/s)? Definitely you should try to filter and collect only needed event not all from those logs.

r. Ismo

View solution in original post

inventsekar
Ultra Champion

30-60 minutes ,...is a too much delay. It should not take so long. even 3mins delay itself is unacceptable.

on the Splunk Cloud side, there should not be much of a delay. The delay is only should be on your systems/network side. 

pls troubleshoot the networks/connections, etc and update us how it goes.. 

0 Karma

isoutamo
SplunkTrust
SplunkTrust
If this is a very busy DC there could be hundreds/thousands events per second. With this amount of event and badly designed collection it cannot work. I have seen same kind of situations couple of time where they couldn’t found any suitable solutions to collect all needed events.
0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

what I have learn/heard is that use WMI only if you haven't any other options and you must collect those logs! It's much better to use UF on those nodes!

As you have 12 DC nodes you probably have quite high event count (1-X kilo events/s)? Definitely you should try to filter and collect only needed event not all from those logs.

r. Ismo

Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...