What is the best practices to collect data (high frequently) out of the Azure Monitor?

[leb7abt]

Hi everyone :),

at the moment i am building a service based on Azure Cloud Infrastructure. I am not very happy with the monitoring solutions given by Microsoft Azure like azure App Insights concerning performance and usability of the dashboards... What i came up with is using Azure Monitor to collect diagnostic logs, metrics from my resources e.g. sql databases , storage blobs ( no app-service, because its at the moment not supported to collect these logs via Azure Monitor) .. Now i would love to know how i can get this data near-realtime into splunk. I already did some research and found "mainly" two solutions.

1. From Azure Monitor directly to an Event Hub to a binded Azure Function which sends the log data via HEC into splunk. Described here: https://github.com/sebastus/AzureFunctionForSplunkCSX

2. From Azure Monitor directly to an Azure blobs/table storage and then periodically via Splunk Add-on for Microsoft Cloud Services into splunk.

Solution 1: I mainly don't like the fact that i need an extra function to send data to the HEC. I would prefer to directly speak to the EventHub via amqp. I know that this is possible but i didn't found a let's call it "trusted add-on" for splunk and i don't want to write it on my own.

Solution 2: I am not quite sure if this is very practicable for my near-realtime needs and don't like the fact that i would have to poll the data and how this would behave on a very huge amount of data (To make sure, i didn't try it ) .

Is there anything i understand wrong or any better way to do this?

Thx for your help!

[larmesto]

This might be helpful for anyone visiting; I have started working on an addon for Azure Event Hubs for Splunk, feel free to use it!
https://splunkbase.splunk.com/app/4343/

regards,

[jconger · Accepted solution]

Microsoft uses 2 main repositories for Azure data (there are APIs that expose different data too, but I'll stick to these):

1. Storage Accounts
2. Event Hubs

Regarding your researched options above:

Option 1 (using an Azure Function to push to Splunk via HEC) is going to get closest to realtime.

Option 2 will get messy as Azure Monitor exposes data in JSON format. If you send that to a blob and have the Splunk Add-on for Microsoft Cloud Services pick it up, several props/transforms will most likely need to re-parse that data if it is a JSON array. Parsing the data can (and has been) done, but it isn't very fun.

You could use the Azure Monitor Add-on for Splunk to pull diagnostic, activity, and metric data from Event Hubs -> https://splunkbase.splunk.com/app/3534/

Or, you could use the Splunk Add-on for Microsoft Cloud Services to pull from storage.

So, to recap:

• Microsoft is going to deliver Azure data to a storage account and/or Event Hub (they aren't mutually exclusive).
• You can use the Splunk Add-on for Microsoft Cloud Services to pull from a storage account.
  
• You can use the Azure Monitor Add-on for Splunk to pull from an Event Hub.
• You can use an Azure Function to push from an Event Hub to Splunk via HEC.