topic Re: Splunk Cloud - Azure Load Balancer Metrics not visible despite correct configuration in All Apps and Add-ons

Splunk Cloud - Azure Load Balancer Metrics not visible despite correct configuration

Leonardo1998 — Tue, 01 Apr 2025 07:24:13 GMT

Hi everyone,
In Splunk Cloud, I am trying to monitor Azure Load Balancer metrics using the .

The Microsoft.Network/loadBalancers namespace has been included in the Azure Metrics input configuration, and from the _internal logs, it seems that the data is being retrieved correctly.

For example, in the logs from mscs_azure_metrics_collector.py, I see the following entry, which suggests that the metrics are being collected:

2025-04-01 07:34:06,096 +0000 log_level=INFO, pid=3442478, tid=ThreadPoolExecutor-0_4, file=mscs_azure_metrics_collector.py, func_name=_index_resource_metrics, code_line_no=526 | Chunked metrics timespan: 2025-04-01T06:28:04Z/2025-04-01T06:33:06Z for resource /subscriptions/<subscripition-id>/resourceGroups/<resource-group-id>/providers/Microsoft.Network/loadBalancers/<load-balancer-name>

However, when I try to find these metrics in Splunk, they are missing. I ran the following query to check the indexes:

| mcatalog values(resource_id) WHERE index=* by resource_id, index, metric_name, namespace

But there is no trace of the Load Balancer. The only namespaces I see are:

microsoft.compute/virtualmachines
microsoft.network/virtualnetworkgateways
microsoft.storage/storageaccounts

Even though the following namespaces have been configured in the input settings:

Microsoft.Compute/virtualMachines
Microsoft.Storage/storageAccounts
Microsoft.Storage/storageAccounts/blobServices
Microsoft.Storage/storageAccounts/fileServices
Microsoft.Storage/storageAccounts/queueServices
Microsoft.Storage/storageAccounts/tableServices
Microsoft.Network/loadBalancers
Microsoft.Network/applicationGateways
Microsoft.Network/virtualNetworkGateways
Microsoft.Network/azureFirewalls

Does anyone know where these metrics might be going?
Is there another way to verify if Splunk is actually indexing Load Balancer metrics?

Thanks in advance for any help!

Re: Splunk Cloud - Azure Load Balancer Metrics not visible despite correct configuration

livehybrid — Tue, 01 Apr 2025 07:56:04 GMT

Hi @Leonardo1998

I have found in the past that sometimes the dimensions extracted by different namespaces can vary slightly.

It might be worth running the following query to try and check that resource_id is available as a dimension to use in your metric searches:

| mcatalog values(_dims) WHERE index=<yourAzureMetricIndex> by namespace

🌟 Did this answer help you? If so, please consider:

Adding kudos to show it was useful
Marking it as the solution if it resolved your issue
Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing.

Re: Splunk Cloud - Azure Load Balancer Metrics not visible despite correct configuration

Leonardo1998 — Tue, 01 Apr 2025 08:11:25 GMT

Hi @livehybrid ,
Thanks for the suggestion.
I tried running the query you suggested, but unfortunately, it didn't make any difference.
The issue still persists.

Re: Splunk Cloud - Azure Load Balancer Metrics not visible despite correct configuration

livehybrid — Tue, 01 Apr 2025 08:45:41 GMT

Hi @Leonardo1998

When you ran that search, did you get a row for your Azure Load Balancer Metrics?

What dimensions did it list?

🌟 Did this answer help you? If so, please consider:

Adding kudos to show it was useful
Marking it as the solution if it resolved your issue
Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing.

Re: Splunk Cloud - Azure Load Balancer Metrics not visible despite correct configuration

Leonardo1998 — Tue, 01 Apr 2025 08:51:13 GMT

Running the search returned exactly the same namespaces as my initial query.
There’s no row related to the Load Balancer metrics.

Re: Splunk Cloud - Azure Load Balancer Metrics not visible despite correct configuration

livehybrid — Tue, 01 Apr 2025 15:31:42 GMT

Hi @Leonardo1998

I think you might need to check the metrics log, the below might help you get started - Ideally we want to see that metrics are being written for your specific source as at the moment it looks like they're not appearing anywhere!

index=_internal group=per_source_thruput series=*azure*

Please can you also confirm that the index for that metric input is definitely as metric index and not an event type index?

🌟 Did this answer help you? If so, please consider:

Adding kudos to show it was useful
Marking it as the solution if it resolved your issue
Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing.

Re: Splunk Cloud - Azure Load Balancer Metrics not visible despite correct configuration

Leonardo1998 — Wed, 02 Apr 2025 08:33:17 GMT

I have tested the query and can confirm that some metrics are being processed, as I mentioned in my initial question. For example:

04-02-2025 08:17:01.787 +0000 INFO Metrics - group=per_source_thruput, ingest_pipe=1, series="/opt/splunk/var/log/splunk/splunk_ta_microsoft-cloudservices_azure_resource.log", kbps="1.382", eps="3.766", kb="41.450", ev=113, avg_age="2.867", max_age=4

However, I still don’t see any logs related to Load Balancers.

Regarding your question: Yes, the index for this metric input is a metric index, not an event index.

That said, I have downloaded the source code of the add-on and I suspect the issue might be on the Azure side. Here’s why:

I debugged the execution flow:

_index_resource_metrics: This function prepares the request. Here, I do see references to the Load Balancer.
_index_metrics: This function calls _fetch_resource_metrics, which sends the request to Azure and processes the results in a loop.
If the response is empty, there is no log message indicating that Azure returned no data.
No relevant errors or exceptions were found in the logs.

Re: Splunk Cloud - Azure Load Balancer Metrics not visible despite correct configuration

livehybrid — Wed, 02 Apr 2025 11:21:14 GMT

Hi @Leonardo1998

Good find - I havent used this app for a while so unsure, but does the input allow you / ask you for a list of operators to apply to the metrics, or even dimensions? I know some of the AWS Cloudwatch metrics ask for this so wondering if its the same.

If so, it could be that these arent quite what its expecting? It sounds like you're on the right track with the debugging - you might need to print out the actual response from the API into the logs so you can see what is being returned!

🌟 Did this answer help you? If so, please consider:

Adding kudos to show it was useful
Marking it as the solution if it resolved your issue
Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing.

Re: Splunk Cloud - Azure Load Balancer Metrics not visible despite correct configuration

Leonardo1998 — Fri, 04 Apr 2025 10:26:24 GMT

Hi, I ended up solving it!

The issue was that Azure isn't sending anything regarding the load balancers. To fix it, I enabled debug logs in the addon, retrieved the details of the call, and simulated the call with Postman.

From the results, Azure returns a 200 status code but with all the "timeseries" fields of the metrics empty.

Thanks for your support!