Hi everyone :),
at the moment i am building a service based on Azure Cloud Infrastructure. I am not very happy with the monitoring solutions given by Microsoft Azure like azure App Insights concerning performance and usability of the dashboards... What i came up with is using Azure Monitor to collect diagnostic logs, metrics from my resources e.g. sql databases , storage blobs ( no app-service, because its at the moment not supported to collect these logs via Azure Monitor) .. Now i would love to know how i can get this data near-realtime into splunk. I already did some research and found "mainly" two solutions.
From Azure Monitor directly to an Event Hub to a binded Azure Function which sends the log data via HEC into splunk. Described here: https://github.com/sebastus/AzureFunctionForSplunkCSX
From Azure Monitor directly to an Azure blobs/table storage and then periodically via Splunk Add-on for Microsoft Cloud Services into splunk.
Solution 1: I mainly don't like the fact that i need an extra function to send data to the HEC. I would prefer to directly speak to the EventHub via amqp. I know that this is possible but i didn't found a let's call it "trusted add-on" for splunk and i don't want to write it on my own.
Solution 2: I am not quite sure if this is very practicable for my near-realtime needs and don't like the fact that i would have to poll the data and how this would behave on a very huge amount of data (To make sure, i didn't try it ) .
Is there anything i understand wrong or any better way to do this?
Thx for your help!
Microsoft uses 2 main repositories for Azure data (there are APIs that expose different data too, but I'll stick to these):
Regarding your researched options above:
Option 1 (using an Azure Function to push to Splunk via HEC) is going to get closest to realtime.
Option 2 will get messy as Azure Monitor exposes data in JSON format. If you send that to a blob and have the Splunk Add-on for Microsoft Cloud Services pick it up, several props/transforms will most likely need to re-parse that data if it is a JSON array. Parsing the data can (and has been) done, but it isn't very fun.
You could use the Azure Monitor Add-on for Splunk to pull diagnostic, activity, and metric data from Event Hubs -> https://splunkbase.splunk.com/app/3534/
Or, you could use the Splunk Add-on for Microsoft Cloud Services to pull from storage.
So, to recap: