All Apps and Add-ons

Can the Azure Monitor Add-on for Splunk run on a HF in AWS

petersonjared
Explorer

We currently have the majority of our infrastructure either on-prem or in AWS with more and more moving to AWS. We do use Azure DevOps though and are looking to get the data from the Azure Monitor Add-on for Splunk into our AWS Splunk environment where our index cluster, search heads, deployment servers, etc reside.

I would prefer not to put the HF in Azure and rather have it in AWS along with the rest of our Splunk environment but I am not seeing any examples or documentation for anyone doing this. I don't see why it would matter where the HF resides as it is polling the event space to get the data. Can this be done? What are the cons of doing so?

Also, regardless of if the HF is in Azure or AWS to run this add-on the HF is a single point of failure. What options are there to mitigate or eliminate that single point of failure (i.e. make it HA-ish) at the HF with the Azure Add-on in play and not duplicate data at the ICs.

0 Karma

petersonjared
Explorer

It looks like you cannot connect to the Event Hub. Do you see the connection leave AWS and come into Azure so you know it isn't being blocked by a Security Group or firewall?

Can you see the Event Hub form your HF outside of using the TA? E.g. I haven't tried it but could you verify the connection from your HF with a curl statement like what is found on:
https://gist.github.com/ivanignatiev/d8fca4ff8b3f729f7337

0 Karma

petersonjared
Explorer

We have this working from a HF in AWS pulling from an Event Hub in Azure. So this can work just fine.

0 Karma

bnakkella
New Member

Hi @petersonjared we are trying to do the same setup HWF on AWS pulling logs from Azure but we are seeing the following errors. Can you please advise

Sample error logs
04-16-2020 03:52:21.077 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" Modular input azure_diagnostic_logs://xxxAzureDiagnostic No connection on hub: insights-logs-auditlogs. Is there a network route to the endpoint?
host = xxxxxhf2 source = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd

04-16-2020 03:52:21.077 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" Modular input azure_diagnostic_logs://xxxAzureDiagnostic No connection on hub: insights-logs-diagnostics. Is there a network route to the endpoint?
host = xxxxxhf2 source = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd

04-16-2020 03:52:20.592 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" Modular input azure_activity_log://xxxAzureMonitorActivity No connection on hub: insights-operational-logs. Is there a network route to the endpoint?
host = xxxxxhf2 source = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd

04-16-2020 03:52:15.739 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" ImportError: No module named splunklib.client

0 Karma

petersonjared
Explorer

Thank you for the reply. We currently do not have any infrastructure running in Azure so it is worth it to at least see what the network traffic cost is initially versus building a stand-alone VM in Azure. Is there anything unique I need to configure to have the HWF from AWS reach into Azure? I have the event Hub in Azure built as per the two fantastic documents below but I don't see what ports and protocols need opened and which direction those connections are instantiated from so I can setup Firewall/security groups/etc to come from AWS and pull from Azure.

https://www.splunk.com/blog/2018/04/20/splunking-microsoft-azure-monitor-data-part-1-azure-setup.htm...
https://www.splunk.com/blog/2018/05/07/splunking-microsoft-azure-monitor-data-part-2-splunk-setup.ht...

jawaharas
Motivator

Answer to this question will be helpful.

However, @petersonjared , Can you raise this query as a separate question?

0 Karma

jconger
Splunk Employee
Splunk Employee

Short answer = "yes" you can run the Azure Monitor add-on in AWS.

There are 3 inputs in the Azure Monitor Add-on:
1) Activity Logs
2) Diagnostic Logs
3) Metrics

Activity logs and diagnostic logs are retrieved from an Event Hub in Azure. Once events are on a hub, the events can be pushed from the hub to Splunk (via Azure Functions and HEC), or the events can be pulled from the hub (via the Azure Monitor Add-on).

One of the big pros of using a HWF inside of Azure to send data outside Azure is the Splunk-to-Splunk compression. This is important to help with data egress costs. But, you have to balance this with running a VM in Azure.

Regarding the single point of failure issue - the Azure Monitor Add-on uses check points to keep up where it is on the event hub. If the HWF goes down (even for days), it will pick up any new events from where it left off when it comes back up.

Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!