topic Re: Azure Event Hub and Add-on for Microsoft Cloud Services Configuration Issues in Getting Data In

Azure Event Hub and Add-on for Microsoft Cloud Services Configuration Issues

WumboJumbo675 — Fri, 08 Mar 2024 16:30:01 GMT

Hello all -

Trying to get Azure Event Hub data to flow into Splunk. Having issues configuring it with the add-on for Microsoft Cloud Services.

I have configured an app in Azure that has Reader & Event Hub Receiver roles. Event Hub has been configured it receive various audit information.

I am trying to configure the input. But receive error message in splunk_ta_microsoft_cloudservices_mscs_azure_event_hub_XYZ.log

Error - 2024-03-08 16:20:31,313 level=ERROR pid=22008 tid=MainThread logger=modular_inputs.mscs_azure_event_hub pos=mscs_azure_event_hub.py:run:939 | datainput="PFG-AzureEventHub1" start_time=1709914805 | message="Error occurred while connecting to eventhub: CBS Token authentication failed. Status code: None Error: client-error CBS Token authentication failed. Status code: None"

I then tried to input the Connection string-primary key in the FQDN space, but receive the below error message. This is occurring because it is trying to create a ckpt file, but the file path is too long and it contains invalid characters.

2024-03-08 14:41:32,112 level=ERROR pid=34216 tid=MainThread logger=modular_inputs.mscs_azure_event_hub pos=utils.py:wrapper:72 | datainput="PFG-AzureEventHub1" start_time=1709908886 | message="Data input was interrupted by an unhandled exception." Traceback (most recent call last): File "L:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\lib\splunksdc\utils.py", line 70, in wrapper return func(*args, **kwargs) File "L:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\lib\modular_inputs\mscs_azure_event_hub.py", line 933, in run consumer = self._create_event_hub_consumer(workspace, config, credential, proxy) File "L:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\lib\modular_inputs\mscs_azure_event_hub.py", line 851, in _create_event_hub_consumer args.consumer_group, File "L:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\lib\modular_inputs\mscs_azure_event_hub.py", line 238, in open checkpoint = SharedLocalCheckpoint(fullname) File "L:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\lib\modular_inputs\mscs_azure_event_hub.py", line 103, in __init__ self._fd = os.open(fullname, os.O_RDWR | os.O_CREAT) FileNotFoundError: [Errno 2] No such file or directory: 'L:\\Program Files\\Splunk\\var\\lib\\splunk\\modinputs\\mscs_azure_event_hub\\Endpoint=sb://REDACTED.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=REDACTED-insights-activity-logs-$Default.v1.ckpt'

Here is my inputs.conf file for the add-on

[mscs_azure_event_hub://PFG-AzureEventHub1] account = AzureActivity consumer_group = $Default event_hub_name = insights-activity-logs event_hub_namespace = REDACTED.servicebus.windows.net index = azure-activity interval = 300 max_batch_size = 300 max_wait_time = 10 sourcetype = mscs:azure:eventhub use_amqp_over_websocket = 1

I have been stuck on this for the past couple of days. Any advice would be greatly appreciated!

Re: Azure Event Hub and Add-on for Microsoft Cloud Services Configuration Issues

WumboJumbo675 — Fri, 08 Mar 2024 18:09:30 GMT

I am now seeing the following error log

2024-03-08 13:06:35,386 level=ERROR pid=34152 tid=MainThread logger=modular_inputs.mscs_azure_event_hub pos=mscs_azure_event_hub.py:run:939 | datainput="PFG-AzureEventHub" start_time=1709903177 | message="Error occurred while connecting to eventhub: Failed to initiate the connection due to exception: Websocket failed to establish connection: SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1106)') Error condition: ErrorCondition.SocketError Error Description: Failed to initiate the connection due to exception: Websocket failed to establish connection: SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1106)')"

Re: Azure Event Hub and Add-on for Microsoft Cloud Services Configuration Issues

wubwub — Tue, 12 Mar 2024 15:30:15 GMT

We are also having the error below:

Error occurred while connecting to eventhub: CBS Token authentication failed

We were told that Splunk wasn't hitting AZ FW at all. Did you solve that? If so was it a network opening? Please share so other can fix as well.

Re: Azure Event Hub and Add-on for Microsoft Cloud Services Configuration Issues

jconger — Tue, 12 Mar 2024 22:16:53 GMT

Where are you applying the Event Hubs Data Receiver role? I usually apply it at the Subscription level so that any other namespaces created in the same subscription will inherit the necessary permissions. There is a walkthrough here (Step 4) => https://lantern.splunk.com/Data_Descriptors/Microsoft/Getting_started_with_Microsoft_Azure_Event_Hub_data

The SSL error you are getting may be a private certificate in the certificate chain. I have also seen similar issues when a network device injects a private cert in the header in outbound traffic.

Re: Azure Event Hub and Add-on for Microsoft Cloud Services Configuration Issues

jconger — Fri, 05 Apr 2024 15:56:56 GMT

2 things to check.

1 - I've seen instances where firewall devices inject private cert on outbound traffic causing error messages like this. Adding an exception for the Splunk forwarder resolved the issue.

2 - if you are using self-signed or internal certs, you may need to add the cert to the add-on's trust list.

Navigate to $SPLUNK_HOME/etc/apps/Splunk_TA_microsoft-cloudservices/lib/certifi
Edit cacert.pem file
Append the contents of your root certificate to this file
Restart Splunk

Re: Azure Event Hub and Add-on for Microsoft Cloud Services Configuration Issues

WumboJumbo675 — Fri, 05 Apr 2024 16:01:41 GMT

The issue has been resolved. Turned out I fat-fingered the entry in Azure Event Hub Input configuration for the 'Event Hub Name' once that was corrected all errors have been resolved and data is ingesting as expected.