All Apps and Add-ons

What is the best practices to collect data (high frequently) out of the Azure Monitor?

leb7abt
Engager

Hi everyone :),

at the moment i am building a service based on Azure Cloud Infrastructure. I am not very happy with the monitoring solutions given by Microsoft Azure like azure App Insights concerning performance and usability of the dashboards... What i came up with is using Azure Monitor to collect diagnostic logs, metrics from my resources e.g. sql databases , storage blobs ( no app-service, because its at the moment not supported to collect these logs via Azure Monitor) .. Now i would love to know how i can get this data near-realtime into splunk. I already did some research and found "mainly" two solutions.

  1. From Azure Monitor directly to an Event Hub to a binded Azure Function which sends the log data via HEC into splunk. Described here: https://github.com/sebastus/AzureFunctionForSplunkCSX

  2. From Azure Monitor directly to an Azure blobs/table storage and then periodically via Splunk Add-on for Microsoft Cloud Services into splunk.

Solution 1: I mainly don't like the fact that i need an extra function to send data to the HEC. I would prefer to directly speak to the EventHub via amqp. I know that this is possible but i didn't found a let's call it "trusted add-on" for splunk and i don't want to write it on my own.

Solution 2: I am not quite sure if this is very practicable for my near-realtime needs and don't like the fact that i would have to poll the data and how this would behave on a very huge amount of data (To make sure, i didn't try it ) .

Is there anything i understand wrong or any better way to do this?

Thx for your help!

1 Solution

jconger
Splunk Employee
Splunk Employee

Microsoft uses 2 main repositories for Azure data (there are APIs that expose different data too, but I'll stick to these):

  1. Storage Accounts
  2. Event Hubs

Regarding your researched options above:

Option 1 (using an Azure Function to push to Splunk via HEC) is going to get closest to realtime.

Option 2 will get messy as Azure Monitor exposes data in JSON format. If you send that to a blob and have the Splunk Add-on for Microsoft Cloud Services pick it up, several props/transforms will most likely need to re-parse that data if it is a JSON array. Parsing the data can (and has been) done, but it isn't very fun.

You could use the Azure Monitor Add-on for Splunk to pull diagnostic, activity, and metric data from Event Hubs -> https://splunkbase.splunk.com/app/3534/

Or, you could use the Splunk Add-on for Microsoft Cloud Services to pull from storage.

So, to recap:

  • Microsoft is going to deliver Azure data to a storage account and/or Event Hub (they aren't mutually exclusive).
  • You can use the Splunk Add-on for Microsoft Cloud Services to pull from a storage account.
  • You can use the Azure Monitor Add-on for Splunk to pull from an Event Hub.
  • You can use an Azure Function to push from an Event Hub to Splunk via HEC.

View solution in original post

larmesto
Path Finder

This might be helpful for anyone visiting; I have started working on an addon for Azure Event Hubs for Splunk, feel free to use it!
https://splunkbase.splunk.com/app/4343/

regards,

0 Karma

jconger
Splunk Employee
Splunk Employee

Microsoft uses 2 main repositories for Azure data (there are APIs that expose different data too, but I'll stick to these):

  1. Storage Accounts
  2. Event Hubs

Regarding your researched options above:

Option 1 (using an Azure Function to push to Splunk via HEC) is going to get closest to realtime.

Option 2 will get messy as Azure Monitor exposes data in JSON format. If you send that to a blob and have the Splunk Add-on for Microsoft Cloud Services pick it up, several props/transforms will most likely need to re-parse that data if it is a JSON array. Parsing the data can (and has been) done, but it isn't very fun.

You could use the Azure Monitor Add-on for Splunk to pull diagnostic, activity, and metric data from Event Hubs -> https://splunkbase.splunk.com/app/3534/

Or, you could use the Splunk Add-on for Microsoft Cloud Services to pull from storage.

So, to recap:

  • Microsoft is going to deliver Azure data to a storage account and/or Event Hub (they aren't mutually exclusive).
  • You can use the Splunk Add-on for Microsoft Cloud Services to pull from a storage account.
  • You can use the Azure Monitor Add-on for Splunk to pull from an Event Hub.
  • You can use an Azure Function to push from an Event Hub to Splunk via HEC.

View solution in original post

Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!