topic Azure eventhub input error with Splunk Add-on for Microsoft Cloud Services in All Apps and Add-ons

Azure eventhub input error with Splunk Add-on for Microsoft Cloud Services

martinborjesson — Thu, 09 Sep 2021 06:46:53 GMT

Hi,

Im using ver 4.1.5 of the cloud services Add-on on my HF Splunk ver 8.0.9.

I've configured an Azure App Account in the App and a input for collecting Azure Devops Audit data. But im not getting any logs in to Splunk. Im getting below warning message in "splunk_ta_microsoft_cloudservices_mscs_azure_event_hub_AzureDevopsAudit.log"

2021-09-09 08:22:45,926 level=WARNING pid=84608 tid=Thread-2 logger=uamqp.authentication.cbs_auth pos=cbs_auth.py:handle_token:122 | Authentication Put-Token failed. Retries exhausted.

CPU rises to 90% when input is enabled.

Any ideas?

Regards, Martin

Re: Azure eventhub input error with Splunk Add-on for Microsoft Cloud Services

jconger — Mon, 13 Sep 2021 23:24:29 GMT

I've heard of this before, and it was an issue with the "Firewalls and virtual networks" settings in the Networking section on the event hub namespace. The settings were blocking the incoming connection from the Splunk add-on. After allowing the IP address (or CIDR) of the Splunk forwarder, data started coming in.

Reference => https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-ip-filtering

Re: Azure eventhub input error with Splunk Add-on for Microsoft Cloud Services

martinborjesson — Tue, 14 Sep 2021 13:14:50 GMT

@jconger

Thanks!

Firewall/Network settings looks fine. However im seeing a lot of the error below:

2021-09-14 15:06:02,025 level=WARNING pid=25855 tid=Thread-1 logger=azure.eventhub._eventprocessor.event_processor pos=event_processor.py:_do_receive:334 | EventProcessor instance 'xxxxx' of xxxx' partition '0' consumer group '$Default'. An error occurred while receiving. The exception is TypeError('list indices must be integers or slices, not str').

Re: Azure eventhub input error with Splunk Add-on for Microsoft Cloud Services

tarungupta0311 — Wed, 22 Sep 2021 12:02:33 GMT

Any suggestion here.

We are also getting the same error.

Re: Azure eventhub input error with Splunk Add-on for Microsoft Cloud Services

martinborjesson — Fri, 08 Oct 2021 18:18:14 GMT

Hi!

Not sure if this helps you but i found a solution that works for me. We ended up sending our azure devops audit logs to a Log Analytics Workspace and from there we are exporting the data to the eventhub. The MS Add-on for Cloud Services was able to fetch the data from the eventhub.

Re: Azure eventhub input error with Splunk Add-on for Microsoft Cloud Services

santosh_u — Thu, 06 Jan 2022 20:37:05 GMT

Hi,

we are doing a PoC to bring in data from Eventhub to Splunk

We are having the exact same issue on our side. we have whitelisted the splunk heavy forwarders on the Azure side. But we are using private endpoints instead of public as all our eventhubs , namespaces and data ( non-prod and prod) are in the prod zone we had to use this.

Any ideas as to what other permissions we might have to add to our application, account . Note we tried using the web socket and disabling it as well both times we get different error messages:

Error when AMQP over websocket is used:

2022-01-06 15:17:25,838 level=WARNING pid=14122 tid=Thread-2 logger=azure.eventhub._eventprocessor.event_processor pos=event_processor.py:_load_balancing:286 | EventProcessor instance 'xxxxxxxxxxxxxxxxxxxxxx' of eventhub 'xxxxxxxxxxxx-NON-PROD' consumer group 'xxxxxxxxxxxxxxxxx_group'. An error occurred while load-balancing and claiming ownership. The exception is AuthenticationError("The messaging entity 'xxxxxxxxxxxxxxxxx.windows.net/xxxxxxxxxxxxxxxxxxxxNON-PROD' could not be found. To know more visit https://aka.ms/sbResourceMgrExceptions. TrackingId:xxxxxxxxxxxxxxxxx-non-prod.servicebus.windows.net:xxxxxxxxx-NON-PROD, Timestamp:2022-01-06T20:17:30\nCBS Token authentication failed.\nStatus code: 404\nDescription: The messaging entity 'xxxxxxx.servicebus.windows.net/xxxxxxxxxxx-NON-PROD' could not be found. To know more visit https://aka.ms/sbResourceMgrExceptions. TrackingId:xxxxxxxxxxx, SystemTracker:exxxxxxx-non-prod.servicebus.windows.net:xxxx-NON-PROD, Timestamp:2022-01-06T20:17:30"). Retrying after 11.688764392967611 seconds

2022-01-06 15:17:19,420 level=WARNING pid=14122 tid=Thread-2 logger=uamqp.authentication.cbs_auth pos=cbs_auth.py:handle_token:119 | Authentication Put-Token failed. Retries exhausted.

Error when AMQP over websocket is disabled:

2022-01-06 15:16:31,386 level=WARNING pid=3860 tid=Thread-2 logger=azure.eventhub._eventprocessor.event_processor pos=event_processor.py:_load_balancing:286 | EventProcessor instance 'xxxxxxx' of eventhub 'xxxxxxxxxxxx01-NON-PROD' consumer group 'preview_data_consumer_group'. An error occurred while load-balancing and claiming ownership. The exception is ConnectError('Failed to open mgmt link: MgmtOpenStatus.Error\nFailed to open mgmt link: MgmtOpenStatus.Error'). Retrying after 10.389673444151345 seconds

2022-01-06 15:16:06,834 level=WARNING pid=3860 tid=Thread-2 logger=azure.eventhub._eventprocessor.event_processor pos=event_processor.py:_load_balancing:286 | EventProcessor instance 'xxxxxxxxxxxxxxx' of eventhub 'xxxxxxxxxx01-NON-PROD' consumer group 'preview_data_consumer_group'. An error occurred while load-balancing and claiming ownership. The exception is ConnectError('Failed to open mgmt link: MgmtOpenStatus.Error\nFailed to open mgmt link: MgmtOpenStatus.Error'). Retrying after 11.352936212034383 seconds

Re: Azure eventhub input error with Splunk Add-on for Microsoft Cloud Services

tarungupta0311 — Fri, 07 Jan 2022 07:27:58 GMT

In our case, it was a wrong Namespace we were using.

Re: Azure eventhub input error with Splunk Add-on for Microsoft Cloud Services

santosh_u — Fri, 07 Jan 2022 21:09:00 GMT

Thanks That seems to be our problem as well.

We fixed it and now the data is flowing.

Re: Azure eventhub input error with Splunk Add-on for Microsoft Cloud Services

raymondmelendez — Thu, 23 Feb 2023 19:38:43 GMT

I have checked namespace on mine but seeing the same error message. This was working at one time and then it stopped