Azure Event Hubs Simple Grabber: u'eventhub.pysdk': 3 clients failed to start.

[anthonysomerset]

I'm trying to set this app up with an event hub, I have verified on my heavy forwarder box that I can use the Azure example python scripts to receive events from the event hub so I have verified connectivity and credentials are OK:

[9:59:58] (ssh) (SUDO) root@prod-backups:~ # python recieve.py        
Received: <azure.eventhub.common.Offset object at 0x7f7ee2456510>, 0
Received: <azure.eventhub.common.Offset object at 0x7f7ee2456610>, 1
Received 2 messages in 0.0514070987701 seconds

However, when I attempt to connect to the exact same event hub with the same credentials from the Splunk app it does not appear to be connecting.

With logs similar to this:

2019-06-21 10:09:47,140 INFO pid=15675 tid=MainThread file=client.py:run:315 | u'eventhub.pysdk-b7c6c13c': Starting 2 clients
2019-06-21 10:09:47,141 INFO pid=15675 tid=MainThread file=connection.py:_state_changed:178 | Connection 'EHReceiver-ab0eed85-5125-4f09-8ff1-84d364014c4f-partition1' state changed from <ConnectionState.START: 0> to <ConnectionState.START: 0>
2019-06-21 10:09:47,587 INFO pid=15675 tid=MainThread file=connection.py:_state_changed:178 | Connection 'EHReceiver-ab0eed85-5125-4f09-8ff1-84d364014c4f-partition1' state changed from <ConnectionState.START: 0> to <ConnectionState.START: 0>
2019-06-21 10:09:47,790 INFO pid=15675 tid=MainThread file=connection.py:_state_changed:178 | Connection 'EHReceiver-ab0eed85-5125-4f09-8ff1-84d364014c4f-partition1' state changed from <ConnectionState.START: 0> to <ConnectionState.HDR_SENT: 2>
2019-06-21 10:09:47,840 INFO pid=15675 tid=MainThread file=connection.py:_state_changed:178 | Connection 'EHReceiver-ab0eed85-5125-4f09-8ff1-84d364014c4f-partition1' state changed from <ConnectionState.HDR_SENT: 2> to <ConnectionState.HDR_EXCH: 3>
2019-06-21 10:09:47,840 INFO pid=15675 tid=MainThread file=connection.py:_state_changed:178 | Connection 'EHReceiver-ab0eed85-5125-4f09-8ff1-84d364014c4f-partition1' state changed from <ConnectionState.HDR_EXCH: 3> to <ConnectionState.OPEN_SENT: 7>
2019-06-21 10:09:47,891 INFO pid=15675 tid=MainThread file=connection.py:_state_changed:178 | Connection 'EHReceiver-ab0eed85-5125-4f09-8ff1-84d364014c4f-partition1' state changed from <ConnectionState.OPEN_SENT: 7> to <ConnectionState.OPENED: 9>
2019-06-21 10:09:47,992 INFO pid=15675 tid=MainThread file=connection.py:work:260 | CBS for connection 'EHReceiver-ab0eed85-5125-4f09-8ff1-84d364014c4f-partition1' completed opening with status: 0
2019-06-21 10:09:48,042 INFO pid=15675 tid=MainThread file=connection.py:work:260 | Token put complete with result: 0, status: 202, description: 'Accepted', connection: 'EHReceiver-ab0eed85-5125-4f09-8ff1-84d364014c4f-partition1'
2019-06-21 10:09:48,093 INFO pid=15675 tid=MainThread file=receiver.py:on_state_changed:296 | Message receiver 'receiver-link-684250f6-5e18-4efc-9dc6-9e0d584b0597' state changed from <MessageReceiverState.Idle: 0> to <MessageReceiverState.Opening: 1> on connection: 'EHReceiver-ab0eed85-5125-4f09-8ff1-84d364014c4f-partition1'
2019-06-21 10:09:48,143 INFO pid=15675 tid=MainThread file=receiver.py:on_state_changed:296 | Message receiver 'receiver-link-684250f6-5e18-4efc-9dc6-9e0d584b0597' state changed from <MessageReceiverState.Opening: 1> to <MessageReceiverState.Open: 2> on connection: 'EHReceiver-ab0eed85-5125-4f09-8ff1-84d364014c4f-partition1'
2019-06-21 10:09:48,194 WARNING pid=15675 tid=MainThread file=client.py:run:324 | u'eventhub.pysdk-b7c6c13c': 1 clients failed to start.
2019-06-21 10:10:18,465 INFO pid=15675 tid=MainThread file=client.py:stop:339 | u'eventhub.pysdk-b7c6c13c': Stopping 2 clients
2019-06-21 10:10:18,465 INFO pid=15675 tid=MainThread file=receiver.py:on_state_changed:296 | Message receiver 'receiver-link-684250f6-5e18-4efc-9dc6-9e0d584b0597' state changed from <MessageReceiverState.Open: 2> to <MessageReceiverState.Closing: 3> on connection: 'EHReceiver-ab0eed85-5125-4f09-8ff1-84d364014c4f-partition1'
2019-06-21 10:10:18,619 INFO pid=15675 tid=MainThread file=connection.py:_close:130 | Shutting down connection 'EHReceiver-ab0eed85-5125-4f09-8ff1-84d364014c4f-partition1'.
2019-06-21 10:10:18,620 INFO pid=15675 tid=MainThread file=cbs_auth.py:close_authenticator:82 | Shutting down CBS session on connection: 'EHReceiver-ab0eed85-5125-4f09-8ff1-84d364014c4f-partition1'.
2019-06-21 10:10:18,620 INFO pid=15675 tid=MainThread file=cbs_auth.py:close_authenticator:86 | Auth closed, destroying session on connection: 'EHReceiver-ab0eed85-5125-4f09-8ff1-84d364014c4f-partition1'.
2019-06-21 10:10:18,620 INFO pid=15675 tid=MainThread file=cbs_auth.py:close_authenticator:89 | Finished shutting down CBS session on connection: 'EHReceiver-ab0eed85-5125-4f09-8ff1-84d364014c4f-partition1'.
2019-06-21 10:10:18,620 INFO pid=15675 tid=MainThread file=connection.py:_state_changed:178 | Connection 'EHReceiver-ab0eed85-5125-4f09-8ff1-84d364014c4f-partition1' state changed from <ConnectionState.OPENED: 9> to <ConnectionState.END: 13>
2019-06-21 10:10:18,622 INFO pid=15675 tid=MainThread file=connection.py:_close:137 | Connection shutdown complete 'EHReceiver-ab0eed85-5125-4f09-8ff1-84d364014c4f-partition1'.

[anthonysomerset]

googling for the error itself suggests a firewall issue specifically but as mentioned using an example MS python script works fine with same credentials/endpoint - seems some other error or config breaking?

[anthonysomerset]

I'd like to add that since this question was published - this app was archived - app author is obviously not interested in supporting it

[mitchcorbett]

Did you ever find a solution to this issue? We're getting the same thing. We checked the firewall, and it looks correct. Thanks.

[anthonysomerset]

no - i had to go and use the Capture based splunk app for now - i think it comes down to the data coming off the event hub must be in a specific format with some specific fields present that at the very least in my case are not present

[mitchcorbett]

Thanks for responding. Which is the capture app? We tried using the event hubs integrator app from the same author with no luck.

[anthonysomerset]

https://splunkbase.splunk.com/app/4343/#/detail

we did some custom tweaks to not hardcode sourcetype, host and index though so that the inputs config could properly override and we could parse the events correctly

[mitchcorbett]

Great, thanks for the tip. We'll give it another go.