Azure Event Hubs Grabber: Why are events consumed from Event Hub, but no events are indexed?

[torerikhelgesen]

Hi
I have just installed the new Splunk App Azure Event Hubs Grabber and configured it properly (I think).
The log output form splunkd.log (no entries) and the application log (see below) indicate no problem afaik.

The Azure dashboard for the Event Hub shows that events are outgoing and throughput also indicates that events are consumed.
No other consumers.

Any idea on why I cannot find the events indexed, or are there any troubleshooting tips?

Log output from one cycle in ta_azure_event_hubs_grabber_azure_event_hubs.log:

2019-05-27 14:00:24,579 INFO pid=12331 tid=MainThread file=setup_util.py:log_info:114 | Log level is not set, use default INFO
2019-05-27 14:00:24,579 INFO pid=12331 tid=MainThread file=client.py:__init__:144 | u'eventhub.pysdk-946894d4': Created the Event Hub client
2019-05-27 14:00:24,581 INFO pid=12331 tid=MainThread file=client.py:run:315 | u'eventhub.pysdk-946894d4': Starting 1 clients
2019-05-27 14:00:24,583 INFO pid=12331 tid=MainThread file=connection.py:_state_changed:178 | Connection 'EHReceiver-2f35150d-2f23-48b0-acd5-43fbdba8eb76-partition0' state changed from  to 
2019-05-27 14:00:24,712 INFO pid=12331 tid=MainThread file=connection.py:_state_changed:178 | Connection 'EHReceiver-2f35150d-2f23-48b0-acd5-43fbdba8eb76-partition0' state changed from  to 
2019-05-27 14:00:24,914 INFO pid=12331 tid=MainThread file=connection.py:_state_changed:178 | Connection 'EHReceiver-2f35150d-2f23-48b0-acd5-43fbdba8eb76-partition0' state changed from  to 
2019-05-27 14:00:24,965 INFO pid=12331 tid=MainThread file=connection.py:_state_changed:178 | Connection 'EHReceiver-2f35150d-2f23-48b0-acd5-43fbdba8eb76-partition0' state changed from  to 
2019-05-27 14:00:24,965 INFO pid=12331 tid=MainThread file=connection.py:_state_changed:178 | Connection 'EHReceiver-2f35150d-2f23-48b0-acd5-43fbdba8eb76-partition0' state changed from  to 
2019-05-27 14:00:25,015 INFO pid=12331 tid=MainThread file=connection.py:_state_changed:178 | Connection 'EHReceiver-2f35150d-2f23-48b0-acd5-43fbdba8eb76-partition0' state changed from  to 
2019-05-27 14:00:25,167 INFO pid=12331 tid=MainThread file=connection.py:work:260 | CBS for connection 'EHReceiver-2f35150d-2f23-48b0-acd5-43fbdba8eb76-partition0' completed opening with status: 0
2019-05-27 14:00:25,217 INFO pid=12331 tid=MainThread file=connection.py:work:260 | Token put complete with result: 0, status: 202, description: 'Accepted', connection: 'EHReceiver-2f35150d-2f23-48b0-acd5-43fbdba8eb76-partition0'
2019-05-27 14:00:25,268 INFO pid=12331 tid=MainThread file=receiver.py:on_state_changed:296 | Message receiver 'receiver-link-cd381d50-7959-42b8-a012-db4cd12a7df9' state changed from  to  on connection: 'EHReceiver-2f35150d-2f23-48b0-acd5-43fbdba8eb76-partition0'
2019-05-27 14:00:25,370 INFO pid=12331 tid=MainThread file=receiver.py:on_state_changed:296 | Message receiver 'receiver-link-cd381d50-7959-42b8-a012-db4cd12a7df9' state changed from  to  on connection: 'EHReceiver-2f35150d-2f23-48b0-acd5-43fbdba8eb76-partition0'
2019-05-27 14:00:25,435 INFO pid=12331 tid=MainThread file=client.py:stop:339 | u'eventhub.pysdk-946894d4': Stopping 1 clients
2019-05-27 14:00:25,436 INFO pid=12331 tid=MainThread file=receiver.py:on_state_changed:296 | Message receiver 'receiver-link-cd381d50-7959-42b8-a012-db4cd12a7df9' state changed from  to  on connection: 'EHReceiver-2f35150d-2f23-48b0-acd5-43fbdba8eb76-partition0'
2019-05-27 14:00:25,713 INFO pid=12331 tid=MainThread file=connection.py:_close:130 | Shutting down connection 'EHReceiver-2f35150d-2f23-48b0-acd5-43fbdba8eb76-partition0'.
2019-05-27 14:00:25,714 INFO pid=12331 tid=MainThread file=cbs_auth.py:close_authenticator:82 | Shutting down CBS session on connection: 'EHReceiver-2f35150d-2f23-48b0-acd5-43fbdba8eb76-partition0'.
2019-05-27 14:00:25,714 INFO pid=12331 tid=MainThread file=cbs_auth.py:close_authenticator:86 | Auth closed, destroying session on connection: 'EHReceiver-2f35150d-2f23-48b0-acd5-43fbdba8eb76-partition0'.
2019-05-27 14:00:25,714 INFO pid=12331 tid=MainThread file=cbs_auth.py:close_authenticator:89 | Finished shutting down CBS session on connection: 'EHReceiver-2f35150d-2f23-48b0-acd5-43fbdba8eb76-partition0'.
2019-05-27 14:00:25,714 INFO pid=12331 tid=MainThread file=connection.py:_state_changed:178 | Connection 'EHReceiver-2f35150d-2f23-48b0-acd5-43fbdba8eb76-partition0' state changed from  to 
2019-05-27 14:00:25,715 INFO pid=12331 tid=MainThread file=connection.py:_close:137 | Connection shutdown complete 'EHReceiver-2f35150d-2f23-48b0-acd5-43fbdba8eb76-partition0'.

[evania]

Hi @torerikhelgesen ,

Did you have a chance to check out any answers? If it worked, please resolve this post by approving it! If your problem is still not solved, keep us updated so that someone else can help you.

Thanks for posting!

[uhaq]

Have you checked your default index?
I was able to get data from multiple partition

For troubleshooting I would suggest looking at the inputs.conf in default and what you have in local to see if any parameter is not defined and try changing the partition #... Its a new app so I would start troubleshooting it as I would start with any other app by checking my configs.

[torerikhelgesen]

Hi. Yes, index is properly configured but events can not be found.. I have checked local and default settings and they seem fine. I have tried to change them and that gives WARNING's in the log.

Any idea if it is possible to change log settings for the add-on to debug?
Other ideas?

Thanks

[uhaq]

Sorry for getting back to you late.

Since the last post a new version of the app (1.0.7) was launched, try installing that and then look for any errors in the Splunkd log?
A feature of the TA is that it will pull data from your eventhub partition if there is anything 'new' written to it . So if your instance is off or restarting there will be a gap in your input (This is assuming that the eventhub is constantly written to). Conversely if the event hub is not busy, there will be no data in your index as the app can only pull in whats recent. This is my understanding so far...

[anthonysomerset]

i set my logging to debug - i see the messages received but not being indexed at all

the authors capture based splunk app works fine but does need tweaking so that we can override source type for each configured input

from my perusal of the this app source code it would appear its expecting event data in a specific format or specific fields to be present that in my case are not - i wonder if this is related