Delivering Events to Splunk via MS Azure Event Hubs - Design Considerations and Recommendations

[TheColorBlack]

Howdy  Splunk Community,

I'm curious if anyone here has any experience, or is currently utilizing Splunk's "Azure Functions for Splunk" , specifically the "event-hubs-hec" solution to successfully push events from their Azure Tenant to their Splunk deployment. If so, I'm ultimately curious what designs / architecture patterns you utilized when deploying and segmenting out your Azure Event Hub Namespaces, and Event Hubs. 

Reading over the README in the repo leads me to believe that you can get away with dumping all of the events generated within your tenant into a single event hub namespace / event hub, assuming you stay within the performance limitations imposed by the event hub.

I don't particularly like this model as I believe it makes troubleshooting ingestion / data issues a bit of a pain since all of your data, regardless of source, or event type is in a single centralized location. so I would like to have a bit more organization than that. 

I'm slowly working on a rough draft of how I think I want to break out my Event Hub Namespaces / Event Hubs but right now I'm not sure if I'm going to make my life, or my development team's life's harder as they will have to interface with this design via Terraform as we continue implementing infrastructure as code in our platform.

My initial breakout looks something like:

- A unique subscription per AZ region we are deployed in, dedicated to logging infrastructure that will contain the Event Hub Namespaces, and corresponding function applications that push events out to Splunk...etc. All infrastructure that exists within a specified region will send their Diagnostic Logging Events (Platform logs / Resource logs) into the logging subscription.

- A EH Namespace for SQL Servers, with EH's broken out per event type generated by the SQL Servers

- An EH Namespace for Keyvaults, with EH's broken out per event type generated by Keyvaults

- An EH namespace for Storage Accounts, with EH's broken out per event type generated by the storage accounts

- An EH namespace for Global Microsoft Services (Azure Active Directory, Microsoft Defender, Sentinel...etc)

- An EH namespace for Azure PaaS / IaaS offerings (Databricks, Azure Data Factory, Cognitive Search...etc)

- An EH namespace for networking events (NAT Gateways, Firewalls, Public IPs, APIM, Frontdoor, WAF...etc)

so on and so forth.

Anyone willing to lend their insight?

[jconger · Accepted solution]

An event hub namespace is going to contain one or more event hubs.  The design pattern I typically see is dumping similar logs into the same hub.

For example, Azure Active Directory sign-ins (including interactive sign-ins, service principal sign-ins, non-interactive sign-ins, etc.) and audit logs go into one hub.  Key Vault logs go into another hub.  Subscription activity goes into another hub - and so on. All of these hubs can be in the same event hub namespace.  The namespace is basically the address of the hubs.

There are some limits to keep in mind.  For example, basic and standard tiers only let you have 10 hubs in the same namespace.  Premium and dedicated tiers have different limits.  Reference: https://learn.microsoft.com/azure/event-hubs/event-hubs-quotas 

You can have multiple namespaces in the same subscription though, and each namespace has its own limits.  So, if you are using a standard event hub tier, and you want to have 11 hubs, you'll need 2 namespaces.  You would deploy the functions to each namespace.

[TheColorBlack]

@jcongerThank you very much for taking the time to respond to my question. I sincerely appreciate it. The path you suggested is likely the one I'm going to continue down.