All Apps and Add-ons

Azure eventhub input error with Splunk Add-on for Microsoft Cloud Services

martinborjesson
Explorer

Hi,

Im using ver 4.1.5 of the cloud services Add-on on my HF Splunk ver 8.0.9.

I've configured an Azure App Account in the App and a input for collecting Azure Devops Audit data. But im not getting any logs in to Splunk. Im getting below warning message in "splunk_ta_microsoft_cloudservices_mscs_azure_event_hub_AzureDevopsAudit.log"

2021-09-09 08:22:45,926 level=WARNING pid=84608 tid=Thread-2 logger=uamqp.authentication.cbs_auth pos=cbs_auth.py:handle_token:122 | Authentication Put-Token failed. Retries exhausted.

CPU rises to 90% when input is enabled.

Any ideas?

 

Regards, Martin

Labels (1)
0 Karma
1 Solution

martinborjesson
Explorer

Hi!

Not sure if this helps you but i found a solution that works for me. We ended up sending our azure devops audit logs to a Log Analytics Workspace and from there we are exporting the data to the eventhub. The MS Add-on for Cloud Services was able to fetch the data from the eventhub.

solution..png

 

 

 

View solution in original post

0 Karma

jconger
Splunk Employee
Splunk Employee

I've heard of this before, and it was an issue with the "Firewalls and virtual networks" settings in the Networking section on the event hub namespace.  The settings were blocking the incoming connection from the Splunk add-on.  After allowing the IP address (or CIDR) of the Splunk forwarder, data started coming in.

Reference => https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-ip-filtering

0 Karma

santosh_u
Engager

Hi,

we are doing a PoC to bring in data from Eventhub to Splunk

We are having the exact same issue on our side. we have whitelisted the splunk heavy forwarders on the Azure side. But we are using private endpoints instead of public as all our eventhubs , namespaces and data ( non-prod and prod) are in the prod zone we had to use this.

Any ideas as to what other permissions we might have to add to our application, account . Note we tried using the web socket and disabling it as well both times we get different error messages:

 

Error when AMQP over websocket is used:

2022-01-06 15:17:25,838 level=WARNING pid=14122 tid=Thread-2 logger=azure.eventhub._eventprocessor.event_processor pos=event_processor.py:_load_balancing:286 | EventProcessor instance 'xxxxxxxxxxxxxxxxxxxxxx' of eventhub 'xxxxxxxxxxxx-NON-PROD' consumer group 'xxxxxxxxxxxxxxxxx_group'. An error occurred while load-balancing and claiming ownership. The exception is AuthenticationError("The messaging entity 'xxxxxxxxxxxxxxxxx.windows.net/xxxxxxxxxxxxxxxxxxxxNON-PROD' could not be found. To know more visit https://aka.ms/sbResourceMgrExceptions. TrackingId:xxxxxxxxxxxxxxxxx-non-prod.servicebus.windows.net:xxxxxxxxx-NON-PROD, Timestamp:2022-01-06T20:17:30\nCBS Token authentication failed.\nStatus code: 404\nDescription: The messaging entity 'xxxxxxx.servicebus.windows.net/xxxxxxxxxxx-NON-PROD' could not be found. To know more visit https://aka.ms/sbResourceMgrExceptions. TrackingId:xxxxxxxxxxx, SystemTracker:exxxxxxx-non-prod.servicebus.windows.net:xxxx-NON-PROD, Timestamp:2022-01-06T20:17:30"). Retrying after 11.688764392967611 seconds

2022-01-06 15:17:19,420 level=WARNING pid=14122 tid=Thread-2 logger=uamqp.authentication.cbs_auth pos=cbs_auth.py:handle_token:119 | Authentication Put-Token failed. Retries exhausted.

 

Error when AMQP over websocket is  disabled:

2022-01-06 15:16:31,386 level=WARNING pid=3860 tid=Thread-2 logger=azure.eventhub._eventprocessor.event_processor pos=event_processor.py:_load_balancing:286 | EventProcessor instance 'xxxxxxx' of eventhub 'xxxxxxxxxxxx01-NON-PROD' consumer group 'preview_data_consumer_group'. An error occurred while load-balancing and claiming ownership. The exception is ConnectError('Failed to open mgmt link: MgmtOpenStatus.Error\nFailed to open mgmt link: MgmtOpenStatus.Error'). Retrying after 10.389673444151345 seconds

2022-01-06 15:16:06,834 level=WARNING pid=3860 tid=Thread-2 logger=azure.eventhub._eventprocessor.event_processor pos=event_processor.py:_load_balancing:286 | EventProcessor instance 'xxxxxxxxxxxxxxx' of eventhub 'xxxxxxxxxx01-NON-PROD' consumer group 'preview_data_consumer_group'. An error occurred while load-balancing and claiming ownership. The exception is ConnectError('Failed to open mgmt link: MgmtOpenStatus.Error\nFailed to open mgmt link: MgmtOpenStatus.Error'). Retrying after 11.352936212034383 seconds

 

 

0 Karma

martinborjesson
Explorer

@jconger 

Thanks!

Firewall/Network settings looks fine. However im seeing a lot of the error below:

 

2021-09-14 15:06:02,025 level=WARNING pid=25855 tid=Thread-1 logger=azure.eventhub._eventprocessor.event_processor pos=event_processor.py:_do_receive:334 | EventProcessor instance 'xxxxx' of xxxx' partition '0' consumer group '$Default'. An error occurred while receiving. The exception is TypeError('list indices must be integers or slices, not str').
0 Karma

tarungupta0311
Explorer

Any suggestion here.

We are also getting the same error.

0 Karma

martinborjesson
Explorer

Hi!

Not sure if this helps you but i found a solution that works for me. We ended up sending our azure devops audit logs to a Log Analytics Workspace and from there we are exporting the data to the eventhub. The MS Add-on for Cloud Services was able to fetch the data from the eventhub.

solution..png

 

 

 

0 Karma

tarungupta0311
Explorer
In our case, it was a wrong Namespace we were using.

santosh_u
Engager

Thanks That seems to be our problem as well.

We fixed it and now the data is flowing.

 

Tags (1)
0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...