Splunk Cloud - Azure Load Balancer Metrics not visible despite correct configuration

[Leonardo1998]

Hi everyone, 
In Splunk Cloud, I am trying to monitor Azure Load Balancer metrics using the Splunk Add-on for Microsoft Cloud Services.

The Microsoft.Network/loadBalancers namespace has been included in the Azure Metrics input configuration, and from the _internal logs, it seems that the data is being retrieved correctly.

For example, in the logs from mscs_azure_metrics_collector.py, I see the following entry, which suggests that the metrics are being collected:

2025-04-01 07:34:06,096 +0000 log_level=INFO, pid=3442478, tid=ThreadPoolExecutor-0_4, file=mscs_azure_metrics_collector.py, func_name=_index_resource_metrics, code_line_no=526 | Chunked metrics timespan: 2025-04-01T06:28:04Z/2025-04-01T06:33:06Z for resource /subscriptions/<subscripition-id>/resourceGroups/<resource-group-id>/providers/Microsoft.Network/loadBalancers/<load-balancer-name>

However, when I try to find these metrics in Splunk, they are missing. I ran the following query to check the indexes:

| mcatalog values(resource_id) WHERE index=* by resource_id, index, metric_name, namespace

But there is no trace of the Load Balancer. The only namespaces I see are:

• microsoft.compute/virtualmachines
• microsoft.network/virtualnetworkgateways
• microsoft.storage/storageaccounts

Even though the following namespaces have been configured in the input settings:

• Microsoft.Compute/virtualMachines
• Microsoft.Storage/storageAccounts
• Microsoft.Storage/storageAccounts/blobServices
• Microsoft.Storage/storageAccounts/fileServices
• Microsoft.Storage/storageAccounts/queueServices
• Microsoft.Storage/storageAccounts/tableServices
• Microsoft.Network/loadBalancers
• Microsoft.Network/applicationGateways
• Microsoft.Network/virtualNetworkGateways
• Microsoft.Network/azureFirewalls

Does anyone know where these metrics might be going?
Is there another way to verify if Splunk is actually indexing Load Balancer metrics?

Thanks in advance for any help!

[livehybrid]

Hi @Leonardo1998 

I have found in the past that sometimes the dimensions extracted by different namespaces can vary slightly.

It might be worth running the following query to try and check that resource_id is available as a dimension to use in your metric searches:

| mcatalog values(_dims) WHERE index=<yourAzureMetricIndex> by namespace

🌟 Did this answer help you? If so, please consider:

• Adding kudos to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing.

[Leonardo1998]

Hi @livehybrid ,
Thanks for the suggestion.
I tried running the query you suggested, but unfortunately, it didn't make any difference.
The issue still persists.

[livehybrid]

Hi @Leonardo1998 

I think you might need to check the metrics log, the below might help you get started - Ideally we want to see that metrics are being written for your specific source as at the moment it looks like they're not appearing anywhere!

index=_internal group=per_source_thruput series=*azure*

Please can you also confirm that the index for that metric input is definitely as metric index and not an event type index?

🌟 Did this answer help you? If so, please consider:

• Adding kudos to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing.

[Leonardo1998]

 I have tested the query and can confirm that some metrics are being processed, as I mentioned in my initial question. For example:

04-02-2025 08:17:01.787 +0000 INFO Metrics - group=per_source_thruput, ingest_pipe=1, series="/opt/splunk/var/log/splunk/splunk_ta_microsoft-cloudservices_azure_resource.log", kbps="1.382", eps="3.766", kb="41.450", ev=113, avg_age="2.867", max_age=4

However, I still don’t see any logs related to Load Balancers.

Regarding your question: Yes, the index for this metric input is a metric index, not an event index.

That said, I have downloaded the source code of the add-on and I suspect the issue might be on the Azure side. Here’s why:

I debugged the execution flow:

1. _index_resource_metrics: This function prepares the request. Here, I do see references to the Load Balancer.

2. _index_metrics: This function calls _fetch_resource_metrics, which sends the request to Azure and processes the results in a loop.

3. If the response is empty, there is no log message indicating that Azure returned no data.

4. No relevant errors or exceptions were found in the logs.

[livehybrid]

Hi @Leonardo1998 

Good find - I havent used this app for a while so unsure, but does the input allow you / ask you for a list of operators to apply to the metrics, or even dimensions? I know some of the AWS Cloudwatch metrics ask for this so wondering if its the same.

If so, it could be that these arent quite what its expecting? It sounds like you're on the right track with the debugging - you might need to print out the actual response from the API into the logs so you can see what is being returned!

🌟 Did this answer help you? If so, please consider:

• Adding kudos to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing.

[Leonardo1998 · Accepted solution]

Hi, I ended up solving it!

The issue was that Azure isn't sending anything regarding the load balancers. To fix it, I enabled debug logs in the addon, retrieved the details of the call, and simulated the call with Postman.

From the results, Azure returns a 200 status code but with all the "timeseries" fields of the metrics empty.

Thanks for your support!

[livehybrid]

Hi @Leonardo1998 

When you ran that search, did you get a row for your Azure Load Balancer Metrics?

What dimensions did it list?

🌟 Did this answer help you? If so, please consider:

• Adding kudos to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing.

[Leonardo1998]

Running the search returned exactly the same namespaces as my initial query.
There’s no row related to the Load Balancer metrics.