Azure Event Hub and Add-on for Microsoft Cloud Services Configuration Issues

[WumboJumbo675]

Hello all -

Trying to get Azure Event Hub data to flow into Splunk. Having issues configuring it with the add-on for Microsoft Cloud Services.

I have configured an app in Azure that has Reader & Event Hub Receiver roles. Event Hub has been configured it receive various audit information.

I am trying to configure the input. But receive error message in splunk_ta_microsoft_cloudservices_mscs_azure_event_hub_XYZ.log

Error - 2024-03-08 16:20:31,313 level=ERROR pid=22008 tid=MainThread logger=modular_inputs.mscs_azure_event_hub pos=mscs_azure_event_hub.py:run:939 | datainput="PFG-AzureEventHub1" start_time=1709914805 | message="Error occurred while connecting to eventhub: CBS Token authentication failed.
Status code: None
Error: client-error
CBS Token authentication failed.
Status code: None" 

I then tried to input the Connection string-primary key in the FQDN space, but receive the below error message. This is occurring because it is trying to create a ckpt file, but the file path is too long and it contains invalid characters.

2024-03-08 14:41:32,112 level=ERROR pid=34216 tid=MainThread logger=modular_inputs.mscs_azure_event_hub pos=utils.py:wrapper:72 | datainput="PFG-AzureEventHub1" start_time=1709908886 | message="Data input was interrupted by an unhandled exception." 
Traceback (most recent call last):
  File "L:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\lib\splunksdc\utils.py", line 70, in wrapper
    return func(*args, **kwargs)
  File "L:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\lib\modular_inputs\mscs_azure_event_hub.py", line 933, in run
    consumer = self._create_event_hub_consumer(workspace, config, credential, proxy)
  File "L:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\lib\modular_inputs\mscs_azure_event_hub.py", line 851, in _create_event_hub_consumer
    args.consumer_group,
  File "L:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\lib\modular_inputs\mscs_azure_event_hub.py", line 238, in open
    checkpoint = SharedLocalCheckpoint(fullname)
  File "L:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\lib\modular_inputs\mscs_azure_event_hub.py", line 103, in __init__
    self._fd = os.open(fullname, os.O_RDWR | os.O_CREAT)
FileNotFoundError: [Errno 2] No such file or directory: 'L:\\Program Files\\Splunk\\var\\lib\\splunk\\modinputs\\mscs_azure_event_hub\\Endpoint=sb://REDACTED.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=REDACTED-insights-activity-logs-$Default.v1.ckpt'

 Here is my inputs.conf file for the add-on

[mscs_azure_event_hub://PFG-AzureEventHub1]
account = AzureActivity
consumer_group = $Default
event_hub_name = insights-activity-logs
event_hub_namespace = REDACTED.servicebus.windows.net
index = azure-activity
interval = 300
max_batch_size = 300
max_wait_time = 10
sourcetype = mscs:azure:eventhub
use_amqp_over_websocket = 1

 I have been stuck on this for the past couple of days. Any advice would be greatly appreciated!

[jconger]

Where are you applying the Event Hubs Data Receiver role?  I usually apply it at the Subscription level so that any other namespaces created in the same subscription will inherit the necessary permissions.  There is a walkthrough here (Step 4) => https://lantern.splunk.com/Data_Descriptors/Microsoft/Getting_started_with_Microsoft_Azure_Event_Hub...

The SSL error you are getting may be a private certificate in the certificate chain.  I have also seen similar issues when a network device injects a private cert in the header in outbound traffic.

[WumboJumbo675]

I am now seeing the following error log

2024-03-08 13:06:35,386 level=ERROR pid=34152 tid=MainThread logger=modular_inputs.mscs_azure_event_hub pos=mscs_azure_event_hub.py:run:939 | datainput="PFG-AzureEventHub" start_time=1709903177 | message="Error occurred while connecting to eventhub: Failed to initiate the connection due to exception: Websocket failed to establish connection: SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1106)')
Error condition: ErrorCondition.SocketError
 Error Description: Failed to initiate the connection due to exception: Websocket failed to establish connection: SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1106)')" 

[jconger]

2 things to check.

1 - I've seen instances where firewall devices inject private cert on outbound traffic causing error messages like this.  Adding an exception for the Splunk forwarder resolved the issue.

2 - if you are using self-signed or internal certs, you may need to add the cert to the add-on's trust list.

• Navigate to $SPLUNK_HOME/etc/apps/Splunk_TA_microsoft-cloudservices/lib/certifi
• Edit cacert.pem file
• Append the contents of your root certificate to this file
• Restart Splunk

[WumboJumbo675 · Accepted solution]

The issue has been resolved. Turned out I fat-fingered the entry in Azure Event Hub Input configuration for the 'Event Hub Name' once that was corrected all errors have been resolved and data is ingesting as expected.

[wubwub]

We are also having the error below:

Error occurred while connecting to eventhub: CBS Token authentication failed

We were told that Splunk wasn't hitting AZ FW at all. Did you solve that? If so was it a network opening? Please share so other can fix as well.