Getting Data In

What is your average indexing rate from your monitoring console ?

itrimble1
Path Finder

I'm trying to determine if a bottleneck exists in my environment. We ingest about 130Gb a day. Syslog events come through without delay, but Windows Events are delayed anywhere between 1,500 - 5,000 minutes.

A caveat is that our environment is hybrid. We host our indexers in Azure. We have an express route VPN set up and it seems to be artificially low when it comes to write speeds on our index cluster. The express route VPN is rated at 1Gbps.

The indexers drives are rated for up to 7500 iOPS. The Heavy Forwards are on-prem.

We have Windows Events going to 4 Heavy Forwarders (load balanced) then to the Index Cluster (Round Robbin)

alt text

Does this indexing rate seam reasonable ? It's never really gotten above 2Mbs.

yannK
Splunk Employee
Splunk Employee

You see a delay on the windows events, from forwarders. Can you also check the internal logs of those forwarders, it will tell you if the problem is on the forwarding or now. (if internal logs are also delayed, then you have a bottleneck after the forwarder)

If only the Windows events are often delayed at the collection level, because the forwarder may be waiting for the AD to do the objects names resolution.

0 Karma

harsmarvania57
Ultra Champion

Are you hitting network bandwidth on your VPN route ? If yes then are you modifying actual raw events from Windows or syslog servers on Heavy Forwarders ? If you are not modifying actual raw events on Heavy Forwarders then I'll suggest you to use Universal Forwarders as intermediate forwarders instead of Heavy Forwarders, because Heavy Forwarders will parse data and add metadata and other stuffs with raw events which increase traffic on network.

Here is good blog post about Universal vs Heavy Forwarders https://www.splunk.com/blog/2016/12/12/universal-or-heavy-that-is-the-question.html

itrimble1
Path Finder

We are not hitting the bandwidth limitation of the VPN. Still have plenty of room to breath. We have cleaned up Windows events (inputs.conf), removed whitespaces, blacklisted events, etc...

We also tried making the WEC's heavy forwarders. Didn't seem to make much difference.

0 Karma

diogofgm
SplunkTrust
SplunkTrust

The average indexing rate will always depend on your particular infrastructure. How many hosts/UF/HF sourcing data do you have? how many indexing pipelines do you have per indexer? There are alot of questions regarding this topic that probably won't solve you particular problem.

Does the syslog and windows events goes through the same HF to the IDX Layer?
Also what do you mean " 4 Heavy Forwarders (load balanced)"? do you have anything (3rd party) between windows UFs and this HFs?

------------
Hope I was able to help you. If so, some karma would be appreciated.
0 Karma

itrimble1
Path Finder

Windows Events come from Windows Event Collectors with UF's installed.
Two indexing pipelines per indexer.

The syslog events are going also going to the Heavy Forwarders before being indexed.

0 Karma

diogofgm
SplunkTrust
SplunkTrust

Have you checked the WEC part? That might be your bottleneck since you're not having issues with syslog going through the same link.

------------
Hope I was able to help you. If so, some karma would be appreciated.
0 Karma

itrimble1
Path Finder

diogofgm
SplunkTrust
SplunkTrust

WEF/WEC is far from the preferred method of collecting windows events. Its usually way better to have UFs everywhere sending data even using your heavies has intermediate forwarders before hitting azure idxs.

Also do you have the windows TA installed in you HF?

------------
Hope I was able to help you. If so, some karma would be appreciated.
0 Karma

itrimble1
Path Finder

Yes it installed on the HF's

0 Karma
Get Updates on the Splunk Community!

Splunk is Nurturing Tomorrow’s Cybersecurity Leaders Today

Meet Carol Wright. She leads the Splunk Academic Alliance program at Splunk. The Splunk Academic Alliance ...

Part 2: A Guide to Maximizing Splunk IT Service Intelligence

Welcome to the second segment of our guide. In Part 1, we covered the essentials of getting started with ITSI ...

Part 1: A Guide to Maximizing Splunk IT Service Intelligence

As modern IT environments continue to grow in complexity and speed, the ability to efficiently manage and ...