All Apps and Add-ons

Microsoft Azure Add-on for Splunk (TA-MS-AAD) Version 2.0.2 but no data is ingesting from Microsoft Azure Event Hub portal.

Hemnaath
Motivator

Hi I have configured the Event Hub inputs as per the Microsoft Azure Add-on for Splunk documentation to fetch the data from Azure port into Splunk. Below are the input and Configuration details provide in the Splunk Add-on

Configuration --> Account -->
Account name *   --> Azure_test
Client ID *  -->  dXXXXXa-07xxx-4xx0-9d4d-273XXXXXXX  --> Taken from the Azure Application ID 
Client Secret * --> xxxxxxxxxxxxxx    ---> Taken from the Azure Application Secret Key 

Configuration --> Proxy --> No Proxy configured

Configuration --> Logging --> INFO --> Save

Inputs 

[azure_event_hub://Azure_EventHub_Test]
connection_string = ********
consumer_group = $Default
event_hub_name = insights-operational-logs
event_hub_timeout = 5
index = microsoft_azure
interval = 30
max_batch_set_iterations = 100
max_batch_size = 100
number_of_threads = 4
source_type = azure:eventhub
sourcetype = azure:eventhub


In Splunk internal logs, I could see this errors 

index=_internal sourcetype="ta:ms:aad:log" source=*hub*

Error Details:

2020-03-13 08:58:59,391 ERROR pid=19115 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_ms_aad/modinput_wrapper/base_modinput.py", line 127, in stream_events
    self.collect_events(ew)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py", line 92, in collect_events
    input_module.collect_events(self, ew)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/input_module_azure_event_hub.py", line 111, in collect_events
    client = EventHubClient.from_connection_string(connection_string, event_hub_path=event_hub_name, http_proxy=HTTP_PROXY)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/azure/eventhub/client_abstract.py", line 272, in from_connection_string
    address, policy, key, entity = _parse_conn_str(conn_str)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/azure/eventhub/client_abstract.py", line 53, in _parse_conn_str
    raise ValueError("Invalid connection string")
ValueError: Invalid connection string

INFO details 

2020-03-13 08:58:59,391 INFO pid=19115 tid=MainThread file=setup_util.py:log_info:114 | Proxy is not enabled!

2020-03-13 08:58:58,216 INFO pid=19115 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1

I am using Azure Free subscription, using Splunk 8.0.2 Enterprise Trail version installed in the Google Cloud VM ware instance with Unbuntu 18.4 LTS with latest version of Python 3.7.5 version configured as default.

Kindly guide me how to fix this issue and also correct me if any of the inputs details should be changed.

0 Karma

mpras123
New Member

Hey,
I also running into the same issue, where in every other input works but not the Eventshub one, the strange thing is that i don't see any information in the debug logs/error logs for this input. Any assistance is really appreciated.
Regards,
Prasad

0 Karma

kimberlytrayson
Path Finder

hey Prasad,
check "Event Hub Name" in input settings. You have to up Event Hub Name there NOT Namespace name.

0 Karma

Hemnaath
Motivator

hey kimberlytrayson, i had applied the Event Hub Name but still it was not providing any data in splunk, then later i came to know from the Author "Jconger" The event hub input doesn't work on Splunk 8 yet. so if you are using the lower version then it should work.

0 Karma

jconger
Splunk Employee
Splunk Employee

The error message is complaining about the connection string. You can copy/paste directly from the Azure portal.

alt text

Hemnaath
Motivator

thanks for your inputs, but I had actually copied the same and pasted in the splunk -->MicrosoftAzure Add-on--for splunk---->input -->EventHub as shown below.

 [azure_event_hub://Azure_EventHub_Test] 
 connection_string = ********   
 consumer_group = $Default
 event_hub_name = insights-operational-logs
 event_hub_timeout = 5
 index = microsoft_azure
 interval = 30
 max_batch_set_iterations = 100
 max_batch_size = 100
 number_of_threads = 4
 source_type = azure:eventhub
 sourcetype = azure:eventhub

Question:

1) Whether we need to mention about Splunk URL details and splunk login user name and password in Microsoft Azure portal --> AD Application Registration
Reason to ask this question is that How Azure will know where I need to send this information (Event Hub Data).

2) How to copy/paste the pix in splunk answers.com.

0 Karma

jconger
Splunk Employee
Splunk Employee

You do not need an Azure AD App Registration for the Event Hub input. You just need the connection string and event hub name. Since the error message pasted above complained about "Invalid connection string", I would try to copy/paste from the Azure portal again just to make sure the connection string is valid.

0 Karma

Hemnaath
Motivator

okay thanks for the input, I have tried copy/paste the Connection string but still I am unable to see the events in the splunk. These are the INFO message I am getting in Splunk.

index="_internal" sourcetype="ta:ms:aad:log"

2020-03-16 08:54:06,284 INFO pid=29567 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1

2020-03-16 08:54:05,078 INFO pid=29567 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1

2020-03-16 08:54:04,380 INFO pid=29567 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1

2020-03-16 08:54:03,322 INFO pid=29567 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1

2020-03-16 08:54:01,838 INFO pid=29503 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1

As I have mentioned earlier I am using Azure Free subscription, using Splunk 8.0.2 Enterprise Trail version installed in the Google Cloud VM ware instance with Unbuntu 18.4 LTS with latest version of Python 3.7.5 version configured as default.

0 Karma

jconger
Splunk Employee
Splunk Employee

I totally missed the Splunk 8 bit. The event hub input doesn't work on Splunk 8 yet.

0 Karma

Hemnaath
Motivator

oh thanks Jconger. hey one more help from you, as i had mentioned in my previous chat that I have the Azure data in splunk but what kind of reports or visualization I can get out of this data in splunk, I could see Microsoft Azure App Template in splunk base but it is for Splunk Add-on for Microsoft Cloud Services. Is there any other app to play around the visualization/report.

thanks in advance.

0 Karma

Hemnaath
Motivator

Hi Jason Conger, Good Morning once again I have re-configured,
EventHubNamesapce, EventHub,Poilcy name, Client secret key, and all other required prerequisites mentioned in both Splunk & Microsoft Azure documents. But still I am unable to see events in Splunk.

We are getting this information for the **Eventhub and Metric inputs** in Splunk. 

index="_internal" sourcetype="ta:ms:aad:log"

2020-03-16 13:18:46,617 INFO pid=25717 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1

2020-03-16 13:18:42,702 INFO pid=25640 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1

**For Active directory audit inputs:**

2020-03-16 13:18:44,734 ERROR pid=25640 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_ms_aad/modinput_wrapper/base_modinput.py", line 127, in stream_events
    self.collect_events(ew)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_audit.py", line 84, in collect_events
    input_module.collect_events(self, ew)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/input_module_MS_AAD_audit.py", line 74, in collect_events
    audit_events = azutils.get_items(helper, access_token, url, items=[])
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_azure_utils/utils.py", line 33, in get_items
    raise e
HTTPError: 401 Client Error: Unauthorized for url: https://graph.microsoft.com/beta/auditLogs/directoryAudits?$orderby=activityDateTime&$filter=activit...

Referred following links to configure the prerequisites in Azure side and splunk

https://docs.microsoft.com/en-us/azure/active-directory/reports-monitoring/tutorial-azure-monitor-st...

https://docs.microsoft.com/en-us/azure/active-directory/reports-monitoring/tutorial-azure-monitor-st...

https://www.splunk.com/en_us/blog/cloud/splunking-microsoft-azure-monitor-data-part-2-splunk-setup.h...

Kindly guide me where is the issue, as I had configured exactly the steps mentioned in the documents.

thanks in advance.

0 Karma

jconger
Splunk Employee
Splunk Employee

The 401 Client error you are seeing is for the Azure AD audit input instead of the Event Hub input. It is stating that the Azure AD application registration used to make the connection from Splunk to Azure does not have the appropriate permissions. Refer to this spreadsheet for the necessary input permissions needed in Azure -> http://bit.ly/Splunk_Azure_Permissions

0 Karma

Hemnaath
Motivator

Hi jconger, I have given the below required permission in Azure portal for Eventhub and for the Audit logs.

    Azure side: 

         Home --> EventHubs --> Eventhubname --> shared access policies --> SAS Policy: RootManageSharedAccesskey --> claims --> Manage,send,Listen 

            Home --> Subscription --> Free Trial | Access control(IAM) --> Role assignments 
            Network Contributor, Reader, Security Admin, Security Reader

Splunk Side configuration details:
Eventhub(Preview) Inputs:
EventHub name: HXXXXXXXX
Connection String -->  Endpoint=sb://Hxxxxxxx-01.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=xxxxxxxxxxxxxxxxxxxxxxxxxx=

Metric(Preview)Inputs:
Azure App Account : Azure_Mon
Tenant ID:axxxx0c-f2xxx-47b8-9xxx3-75xxx708xxx4
Subscription ID: xxxxxc1a-xxxx-4ac1-8xxx4-f7fxxxxxxx90

Microsoft Azure Active Directory Audit:
Azure App Account : Azure_Mon
Tenant ID:axxxx0c-f2xxx-47b8-9xxx3-75xxx708xxx4

After configuring the above inputs and permission, still no luck i am seeing this below error in splunk internal logs

index="_internal" sourcetype="ta:ms:aad:log"  connectionpool.py:_new_conn:758

2020-03-19 18:03:48,706 INFO pid=9135 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1

index="_internal" sourcetype="ta:ms:aad:log" file="setup_util.py:log_info:114"

2020-03-19 18:49:49,320 INFO pid=29789 tid=MainThread file=setup_util.py:log_info:114 | Proxy is not enabled!


index="_internal" sourcetype="ta:ms:aad:log" file="setup_util.py:log_info:114 | Log level is not set"

2020-03-19 18:05:25,956 INFO pid=9953 tid=MainThread file=setup_util.py:log_info:114 | Log level is not set, use default INFO

index="_internal" sourcetype="ta:ms:aad:log" file="client_abstract.py:__init__:161"

2020-03-19 18:50:59,195 INFO pid=30328 tid=MainThread file=client_abstract.py:__init__:161 | u'eventhub.pysdk-07a5fdec': Created the Event Hub client

index="_internal" sourcetype="ta:ms:aad:log" file="splunk_rest_client.py:_request_handler:100"

2020-03-19 18:50:45,261 INFO pid=30224 tid=MainThread file=splunk_rest_client.py:_request_handler:100 | Use HTTP connection pooling

index="_internal" sourcetype="ta:ms:aad:log" file="base_modinput.py:log_error:307"

2020-03-19 18:53:14,176 ERROR pid=31612 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_ms_aad/modinput_wrapper/base_modinput.py", line 127, in stream_events
    self.collect_events(ew)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_audit.py", line 84, in collect_events
    input_module.collect_events(self, ew)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/input_module_MS_AAD_audit.py", line 74, in collect_events
    audit_events = azutils.get_items(helper, access_token, url, items=[])
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_azure_utils/utils.py", line 33, in get_items
    raise e
HTTPError: 401 Client Error: Unauthorized for url: https://graph.microsoft.com/beta/auditLogs/directoryAudits?$orderby=activityDateTime&$filter=activit...

Kindly guide me how to fix this issue, we are finding it difficult to fix this issue.

0 Karma

jconger
Splunk Employee
Splunk Employee

Here is a screenshot of what the permissions on the application should be. Also, be sure to click the "Grant admin consent..." button.

alt text

0 Karma

Hemnaath
Motivator

Hi jconger thanks for sharing the screen shot, I would like to share my steps with the screen shot, So is there any other way I can share the details with you apart from answers.com as I am not sure how to attach the screen in the answers.com.

0 Karma

jconger
Splunk Employee
Splunk Employee

You should be able to post screen shots by uploading them somewhere publicly accessible and then using Markdown to include them in your post. Something like this:

![alt text][1]

  [1]: https://cdn.apps.splunk.com/media/public/docimages/6245b8de-6a2e-11ea-ab4f-0a356b057c84.png

But, feel free to send me an email also - jconger (at) splunk (dot) com

0 Karma

Hemnaath
Motivator

Hi Jconger, Good Morning, I had sent you the email with the steps for both Azure and Splunk configuration, Kindly do guide me to fix the issue.

0 Karma

Hemnaath
Motivator

Hi Jconger, Good Morning, I am still facing an issue in ingesting the Azure Event Hub details in splunk. Regarding this I had sent you mail with the configuration steps and error details screen shots. Kindly guide me how to troubleshoot and fix the issue.

0 Karma

Hemnaath
Motivator

Hi Jconger, I was able to ingest most of the Microsoft azure inputs into splunk, thanks for guiding to resolve the issue.

But still I am not sure what kind of out put we will be able to see when we ingest Azure Event Hub inputs in Splunk. As I am unable to see any output for this input alone.

Now I have the Azure data in splunk but what kind of visualization I can get out of this data in splunk, I could see Microsoft Azure App Template in splunk base but it is for Splunk Add-on for Microsoft Cloud Services. So is there any other way I can get some report/dashboard using Microsoft Azure add-on for splunk.

0 Karma

Hemnaath
Motivator

I am getting same error not sure where is the problem so could anyone guide me in resolving this issue.

index="_internal" sourcetype="ta:ms:aad:log" source="/opt/splunk/var/log/splunk/ta_ms_aad_MS_AAD_audit.log"

2020-03-16 17:40:26,976 ERROR pid=5422 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_ms_aad/modinput_wrapper/base_modinput.py", line 127, in stream_events
    self.collect_events(ew)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_audit.py", line 84, in collect_events
    input_module.collect_events(self, ew)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/input_module_MS_AAD_audit.py", line 74, in collect_events
    audit_events = azutils.get_items(helper, access_token, url, items=[])
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_azure_utils/utils.py", line 33, in get_items
    raise e
HTTPError: 401 Client Error: Unauthorized for url: https://graph.microsoft.com/beta/auditLogs/directoryAudits?$orderby=activityDateTime&$filter=activit...

index="_internal" sourcetype="ta:ms:aad:log" source="/opt/splunk/var/log/splunk/ta_ms_aad_azure_event_hub.log"

2020-03-16 17:40:25,196 INFO pid=5444 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1


index="_internal" sourcetype="ta:ms:aad:log" source="/opt/splunk/var/log/splunk/ta_ms_aad_azure_metrics.log"

2020-03-16 17:46:15,387 INFO pid=8584 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...