All Apps and Add-ons
Highlighted

Microsoft Azure Add-on Error setting/pulling Event Hub

Communicator

Hello,

I'm trying to integrate with Event Hub

I've entered my Connection String:

Endpoint=sb://<event-hub>.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=<shared-access-key>

And EventHub Name

But getting the following error that I'm not sure what it means

2019-10-30 15:11:50,000 ERROR pid=30610 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_ms_aad/modinput_wrapper/base_modinput.py", line 127, in stream_events
    self.collect_events(ew)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py", line 92, in collect_events
    input_module.collect_events(self, ew)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/input_module_azure_event_hub.py", line 112, in collect_events
    partition_ids = client.get_partition_ids()
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/azure/eventhub/client.py", line 163, in get_partition_ids
    return self.get_properties()['partition_ids']
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/azure/eventhub/client.py", line 146, in get_properties
    response = self._management_request(mgmt_msg, op_type=b'com.microsoft:eventhub')
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/azure/eventhub/client.py", line 127, in _management_request
    self._handle_exception(exception, retry_count, max_retries)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/azure/eventhub/client.py", line 105, in _handle_exception
    _handle_exception(exception, retry_count, max_retries, self)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/azure/eventhub/error.py", line 196, in _handle_exception
    raise error
ConnectError: Unable to open management session. Please confirm URI namespace exists.
Unable to open management session. Please confirm URI namespace exists.
Highlighted

Re: Microsoft Azure Add-on Error setting/pulling Event Hub

Motivator

I've seen something similar where the event hub namespace was specified in the input instead of the event hub name.

alt text

0 Karma
Highlighted

Re: Microsoft Azure Add-on Error setting/pulling Event Hub

Communicator

That doesn't seem to be the problem 😞

Here are more errors from running index=_internal sourcetype="ta:ms:aad:log" source=*hub*

2019-10-30 15:54:47,997 INFO pid=44249 tid=MainThread file=mgmt_operation.py:__init__:65 | 'Failed opening message receiver' ('/data/src/vendor/azure-uamqp-c/src/amqp_management.c':'amqp_management_open_async':981)
2019-10-30 15:54:47,997 INFO pid=44249 tid=MainThread file=mgmt_operation.py:__init__:65 | 'Link attach failed' ('/data/src/vendor/azure-uamqp-c/src/message_receiver.c':'messagereceiver_open':362)
2019-10-30 15:54:47,997 INFO pid=44249 tid=MainThread file=mgmt_operation.py:__init__:65 | 'Begin session failed' ('/data/src/vendor/azure-uamqp-c/src/link.c':'link_attach':1154)
2019-10-30 15:54:47,996 INFO pid=44249 tid=MainThread file=connection.py:_state_changed:181 | Connection with ID '8a0fbd26-fffa-4ace-993b-455aef8737bd' unexpectedly in an error state. Closing: False, Error: None
2019-10-30 15:54:47,996 INFO pid=44249 tid=MainThread file=connection.py:_state_changed:177 | Connection '8a0fbd26-fffa-4ace-993b-455aef8737bd' state changed from <ConnectionState.START: 0> to <ConnectionState.END: 13>
2019-10-30 15:54:47,996 INFO pid=44249 tid=MainThread file=mgmt_operation.py:__init__:65 | 'Opening the underlying IO failed' ('/data/src/vendor/azure-uamqp-c/src/connection.c':'connection_open':1344)
2019-10-30 15:54:47,996 INFO pid=44249 tid=MainThread file=mgmt_operation.py:__init__:65 | 'xio_open failed' ('/data/src/vendor/azure-uamqp-c/src/saslclientio.c':'saslclientio_open_async':1097)
2019-10-30 15:54:47,996 INFO pid=44249 tid=MainThread file=mgmt_operation.py:__init__:65 | 'Failed opening the underlying I/O.' ('/data/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/tlsio_openssl.c':'tlsio_openssl_open':1258)
2019-10-30 15:54:47,996 INFO pid=44249 tid=MainThread file=mgmt_operation.py:__init__:65 | 'Invalid tlsio_state. Expected state is TLSIO_STATE_OPENING_UNDERLYING_IO.' ('/data/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/tlsio_openssl.c':'on_underlying_io_open_complete':760)
2019-10-30 15:54:47,996 INFO pid=44249 tid=MainThread file=mgmt_operation.py:__init__:65 | 'Closing tlsio from a state other than TLSIO_STATE_EXT_OPEN or TLSIO_STATE_EXT_ERROR'
2019-10-30 15:54:47,995 INFO pid=44249 tid=MainThread file=mgmt_operation.py:__init__:65 | 'wait_for_connection failed' ('/data/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/socketio_berkeley.c':'socketio_open':770)
2019-10-30 15:54:47,989 INFO pid=44249 tid=MainThread file=connection.py:_state_changed:177 | Connection '8a0fbd26-fffa-4ace-993b-455aef8737bd' state changed from <ConnectionState.START: 0> to <ConnectionState.START: 0>
2019-10-30 15:54:47,988 INFO pid=44249 tid=MainThread file=connection.py:_close:136 | Connection shutdown complete 'fda99a3d-6f6a-44d1-8921-8bf1d235fb95'.
2019-10-30 15:54:47,988 INFO pid=44249 tid=MainThread file=connection.py:_close:129 | Shutting down connection 'fda99a3d-6f6a-44d1-8921-8bf1d235fb95'.
2019-10-30 15:54:47,988 INFO pid=44249 tid=MainThread file=error.py:_handle_exception:233 | u'eventhub.pysdk-5ec316b9' has an exception (AMQPConnectionError('Unable to open management session. Please confirm URI namespace exists.',)). Retrying...
2019-10-30 15:54:47,187 INFO pid=44249 tid=MainThread file=mgmt_operation.py:__init__:65 | 'Failed opening message receiver' ('/data/src/vendor/azure-uamqp-c/src/amqp_management.c':'amqp_management_open_async':981)
2019-10-30 15:54:47,187 INFO pid=44249 tid=MainThread file=mgmt_operation.py:__init__:65 | 'Link attach failed' ('/data/src/vendor/azure-uamqp-c/src/message_receiver.c':'messagereceiver_open':362)
2019-10-30 15:54:47,186 INFO pid=44249 tid=MainThread file=mgmt_operation.py:__init__:65 | 'Begin session failed' ('/data/src/vendor/azure-uamqp-c/src/link.c':'link_attach':1154)
2019-10-30 15:54:47,186 INFO pid=44249 tid=MainThread file=connection.py:_state_changed:181 | Connection with ID 'fda99a3d-6f6a-44d1-8921-8bf1d235fb95' unexpectedly in an error state. Closing: False, Error: None
2019-10-30 15:54:47,186 INFO pid=44249 tid=MainThread file=connection.py:_state_changed:177 | Connection 'fda99a3d-6f6a-44d1-8921-8bf1d235fb95' state changed from <ConnectionState.START: 0> to <ConnectionState.END: 13>
2019-10-30 15:54:47,186 INFO pid=44249 tid=MainThread file=mgmt_operation.py:__init__:65 | 'Opening the underlying IO failed' ('/data/src/vendor/azure-uamqp-c/src/connection.c':'connection_open':1344)
2019-10-30 15:54:47,186 INFO pid=44249 tid=MainThread file=mgmt_operation.py:__init__:65 | 'xio_open failed' ('/data/src/vendor/azure-uamqp-c/src/saslclientio.c':'saslclientio_open_async':1097)
2019-10-30 15:54:47,186 INFO pid=44249 tid=MainThread file=mgmt_operation.py:__init__:65 | 'Failed opening the underlying I/O.' ('/data/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/tlsio_openssl.c':'tlsio_openssl_open':1258)
2019-10-30 15:54:47,186 INFO pid=44249 tid=MainThread file=mgmt_operation.py:__init__:65 | 'Invalid tlsio_state. Expected state is TLSIO_STATE_OPENING_UNDERLYING_IO.' ('/data/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/tlsio_openssl.c':'on_underlying_io_open_complete':760)
2019-10-30 15:54:47,185 INFO pid=44249 tid=MainThread file=mgmt_operation.py:__init__:65 | 'Closing tlsio from a state other than TLSIO_STATE_EXT_OPEN or TLSIO_STATE_EXT_ERROR'
2019-10-30 15:54:47,185 INFO pid=44249 tid=MainThread file=mgmt_operation.py:__init__:65 | 'wait_for_connection failed' ('/data/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/socketio_berkeley.c':'socketio_open':770)
2019-10-30 15:54:47,178 INFO pid=44249 tid=MainThread file=connection.py:_state_changed:177 | Connection 'fda99a3d-6f6a-44d1-8921-8bf1d235fb95' state changed from <ConnectionState.START: 0> to <ConnectionState.START: 0>
2019-10-30 15:54:47,176 INFO pid=44249 tid=MainThread file=client_abstract.py:__init__:161 | u'eventhub.pysdk-5ec316b9': Created the Event Hub client
2019-10-30 15:54:47,175 INFO pid=44249 tid=MainThread file=setup_util.py:log_info:114 | Proxy is not enabled!
2019-10-30 15:54:46,241 INFO pid=44249 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-30 15:54:45,295 INFO pid=44249 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-30 15:54:44,749 INFO pid=44249 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-30 15:54:44,068 INFO pid=44249 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-30 15:54:42,763 ERROR pid=44198 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_ms_aad/modinput_wrapper/base_modinput.py", line 127, in stream_events
    self.collect_events(ew)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py", line 92, in collect_events
    input_module.collect_events(self, ew)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/input_module_azure_event_hub.py", line 112, in collect_events
    partition_ids = client.get_partition_ids()
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/azure/eventhub/client.py", line 163, in get_partition_ids
    return self.get_properties()['partition_ids']
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/azure/eventhub/client.py", line 146, in get_properties
    response = self._management_request(mgmt_msg, op_type=b'com.microsoft:eventhub')
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/azure/eventhub/client.py", line 127, in _management_request
    self._handle_exception(exception, retry_count, max_retries)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/azure/eventhub/client.py", line 105, in _handle_exception
    _handle_exception(exception, retry_count, max_retries, self)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/azure/eventhub/error.py", line 196, in _handle_exception
    raise error
ConnectError: Unable to open management session. Please confirm URI namespace exists.
Unable to open management session. Please confirm URI namespace exists.
2019-10-30 15:54:42,762 INFO pid=44198 tid=MainThread file=connection.py:_close:136 | Connection shutdown complete 'c0b76f33-cac3-44bc-b6b2-a7798c8d363f'.
2019-10-30 15:54:42,762 INFO pid=44198 tid=MainThread file=connection.py:_close:129 | Shutting down connection 'c0b76f33-cac3-44bc-b6b2-a7798c8d363f'.
0 Karma
Highlighted

Re: Microsoft Azure Add-on Error setting/pulling Event Hub

Communicator

I just noticed this Add-on is for Ubuntu or Darwin systems, what would I need for CentOS? Any particular Python libs?

0 Karma
Highlighted

Re: Microsoft Azure Add-on Error setting/pulling Event Hub

Motivator

The error is pointing to a namespace issue. I looked at your connection string, it it should be:

Endpoint=sb://<event-hub-namespace>.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=<shared-access-key>

0 Karma
Highlighted

Re: Microsoft Azure Add-on Error setting/pulling Event Hub

Communicator

The problem was that TCP ports 5671 and 5672 were blocked from the Internal firewall out to the Internet.

“... data inputs use AMQP to connect to event hub over TLS using ports 5671 / 5672 as described in the AMQP 1.0 Service Bus and Event Hubs protocol guide. So, if you are having connection/authentication issues, check that these ports are open on your Splunk instance.”

The network administrator just allowed connections to these ports and the data started coming in!!

Highlighted

Re: Microsoft Azure Add-on Error setting/pulling Event Hub

Motivator

Great news - thanks for posting the solution for posterity!