- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello all -
Trying to get Azure Event Hub data to flow into Splunk. Having issues configuring it with the add-on for Microsoft Cloud Services.
I have configured an app in Azure that has Reader & Event Hub Receiver roles. Event Hub has been configured it receive various audit information.
I am trying to configure the input. But receive error message in splunk_ta_microsoft_cloudservices_mscs_azure_event_hub_XYZ.log
Error - 2024-03-08 16:20:31,313 level=ERROR pid=22008 tid=MainThread logger=modular_inputs.mscs_azure_event_hub pos=mscs_azure_event_hub.py:run:939 | datainput="PFG-AzureEventHub1" start_time=1709914805 | message="Error occurred while connecting to eventhub: CBS Token authentication failed.
Status code: None
Error: client-error
CBS Token authentication failed.
Status code: None"
I then tried to input the Connection string-primary key in the FQDN space, but receive the below error message. This is occurring because it is trying to create a ckpt file, but the file path is too long and it contains invalid characters.
2024-03-08 14:41:32,112 level=ERROR pid=34216 tid=MainThread logger=modular_inputs.mscs_azure_event_hub pos=utils.py:wrapper:72 | datainput="PFG-AzureEventHub1" start_time=1709908886 | message="Data input was interrupted by an unhandled exception."
Traceback (most recent call last):
File "L:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\lib\splunksdc\utils.py", line 70, in wrapper
return func(*args, **kwargs)
File "L:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\lib\modular_inputs\mscs_azure_event_hub.py", line 933, in run
consumer = self._create_event_hub_consumer(workspace, config, credential, proxy)
File "L:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\lib\modular_inputs\mscs_azure_event_hub.py", line 851, in _create_event_hub_consumer
args.consumer_group,
File "L:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\lib\modular_inputs\mscs_azure_event_hub.py", line 238, in open
checkpoint = SharedLocalCheckpoint(fullname)
File "L:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\lib\modular_inputs\mscs_azure_event_hub.py", line 103, in __init__
self._fd = os.open(fullname, os.O_RDWR | os.O_CREAT)
FileNotFoundError: [Errno 2] No such file or directory: 'L:\\Program Files\\Splunk\\var\\lib\\splunk\\modinputs\\mscs_azure_event_hub\\Endpoint=sb://REDACTED.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=REDACTED-insights-activity-logs-$Default.v1.ckpt'
Here is my inputs.conf file for the add-on
[mscs_azure_event_hub://PFG-AzureEventHub1]
account = AzureActivity
consumer_group = $Default
event_hub_name = insights-activity-logs
event_hub_namespace = REDACTED.servicebus.windows.net
index = azure-activity
interval = 300
max_batch_size = 300
max_wait_time = 10
sourcetype = mscs:azure:eventhub
use_amqp_over_websocket = 1
I have been stuck on this for the past couple of days. Any advice would be greatly appreciated!
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The issue has been resolved. Turned out I fat-fingered the entry in Azure Event Hub Input configuration for the 'Event Hub Name' once that was corrected all errors have been resolved and data is ingesting as expected.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content


Where are you applying the Event Hubs Data Receiver role? I usually apply it at the Subscription level so that any other namespaces created in the same subscription will inherit the necessary permissions. There is a walkthrough here (Step 4) => https://lantern.splunk.com/Data_Descriptors/Microsoft/Getting_started_with_Microsoft_Azure_Event_Hub...
The SSL error you are getting may be a private certificate in the certificate chain. I have also seen similar issues when a network device injects a private cert in the header in outbound traffic.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I am now seeing the following error log
2024-03-08 13:06:35,386 level=ERROR pid=34152 tid=MainThread logger=modular_inputs.mscs_azure_event_hub pos=mscs_azure_event_hub.py:run:939 | datainput="PFG-AzureEventHub" start_time=1709903177 | message="Error occurred while connecting to eventhub: Failed to initiate the connection due to exception: Websocket failed to establish connection: SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1106)')
Error condition: ErrorCondition.SocketError
Error Description: Failed to initiate the connection due to exception: Websocket failed to establish connection: SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1106)')"
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content


2 things to check.
1 - I've seen instances where firewall devices inject private cert on outbound traffic causing error messages like this. Adding an exception for the Splunk forwarder resolved the issue.
2 - if you are using self-signed or internal certs, you may need to add the cert to the add-on's trust list.
- Navigate to $SPLUNK_HOME/etc/apps/Splunk_TA_microsoft-cloudservices/lib/certifi
- Edit cacert.pem file
- Append the contents of your root certificate to this file
- Restart Splunk
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The issue has been resolved. Turned out I fat-fingered the entry in Azure Event Hub Input configuration for the 'Event Hub Name' once that was corrected all errors have been resolved and data is ingesting as expected.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We are also having the error below:
Error occurred while connecting to eventhub: CBS Token authentication failed
We were told that Splunk wasn't hitting AZ FW at all. Did you solve that? If so was it a network opening? Please share so other can fix as well.
