Hi I have configured the Event Hub inputs as per the Microsoft Azure Add-on for Splunk documentation to fetch the data from Azure port into Splunk. Below are the input and Configuration details provide in the Splunk Add-on
Configuration --> Account -->
Account name * --> Azure_test
Client ID * --> dXXXXXa-07xxx-4xx0-9d4d-273XXXXXXX --> Taken from the Azure Application ID
Client Secret * --> xxxxxxxxxxxxxx ---> Taken from the Azure Application Secret Key
Configuration --> Proxy --> No Proxy configured
Configuration --> Logging --> INFO --> Save
Inputs
[azure_event_hub://Azure_EventHub_Test]
connection_string = ********
consumer_group = $Default
event_hub_name = insights-operational-logs
event_hub_timeout = 5
index = microsoft_azure
interval = 30
max_batch_set_iterations = 100
max_batch_size = 100
number_of_threads = 4
source_type = azure:eventhub
sourcetype = azure:eventhub
In Splunk internal logs, I could see this errors
index=_internal sourcetype="ta:ms:aad:log" source=*hub*
Error Details:
2020-03-13 08:58:59,391 ERROR pid=19115 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_ms_aad/modinput_wrapper/base_modinput.py", line 127, in stream_events
self.collect_events(ew)
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py", line 92, in collect_events
input_module.collect_events(self, ew)
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/input_module_azure_event_hub.py", line 111, in collect_events
client = EventHubClient.from_connection_string(connection_string, event_hub_path=event_hub_name, http_proxy=HTTP_PROXY)
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/azure/eventhub/client_abstract.py", line 272, in from_connection_string
address, policy, key, entity = _parse_conn_str(conn_str)
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/azure/eventhub/client_abstract.py", line 53, in _parse_conn_str
raise ValueError("Invalid connection string")
ValueError: Invalid connection string
INFO details
2020-03-13 08:58:59,391 INFO pid=19115 tid=MainThread file=setup_util.py:log_info:114 | Proxy is not enabled!
2020-03-13 08:58:58,216 INFO pid=19115 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
I am using Azure Free subscription, using Splunk 8.0.2 Enterprise Trail version installed in the Google Cloud VM ware instance with Unbuntu 18.4 LTS with latest version of Python 3.7.5 version configured as default.
Kindly guide me how to fix this issue and also correct me if any of the inputs details should be changed.
Hey,
I also running into the same issue, where in every other input works but not the Eventshub one, the strange thing is that i don't see any information in the debug logs/error logs for this input. Any assistance is really appreciated.
Regards,
Prasad
hey Prasad,
check "Event Hub Name" in input settings. You have to up Event Hub Name there NOT Namespace name.
hey kimberlytrayson, i had applied the Event Hub Name but still it was not providing any data in splunk, then later i came to know from the Author "Jconger" The event hub input doesn't work on Splunk 8 yet. so if you are using the lower version then it should work.
thanks for your inputs, but I had actually copied the same and pasted in the splunk -->MicrosoftAzure Add-on--for splunk---->input -->EventHub as shown below.
[azure_event_hub://Azure_EventHub_Test]
connection_string = ********
consumer_group = $Default
event_hub_name = insights-operational-logs
event_hub_timeout = 5
index = microsoft_azure
interval = 30
max_batch_set_iterations = 100
max_batch_size = 100
number_of_threads = 4
source_type = azure:eventhub
sourcetype = azure:eventhub
Question:
1) Whether we need to mention about Splunk URL details and splunk login user name and password in Microsoft Azure portal --> AD Application Registration
Reason to ask this question is that How Azure will know where I need to send this information (Event Hub Data).
2) How to copy/paste the pix in splunk answers.com.
You do not need an Azure AD App Registration for the Event Hub input. You just need the connection string and event hub name. Since the error message pasted above complained about "Invalid connection string", I would try to copy/paste from the Azure portal again just to make sure the connection string is valid.
okay thanks for the input, I have tried copy/paste the Connection string but still I am unable to see the events in the splunk. These are the INFO message I am getting in Splunk.
index="_internal" sourcetype="ta:ms:aad:log"
2020-03-16 08:54:06,284 INFO pid=29567 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2020-03-16 08:54:05,078 INFO pid=29567 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2020-03-16 08:54:04,380 INFO pid=29567 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2020-03-16 08:54:03,322 INFO pid=29567 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2020-03-16 08:54:01,838 INFO pid=29503 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
As I have mentioned earlier I am using Azure Free subscription, using Splunk 8.0.2 Enterprise Trail version installed in the Google Cloud VM ware instance with Unbuntu 18.4 LTS with latest version of Python 3.7.5 version configured as default.
I totally missed the Splunk 8 bit. The event hub input doesn't work on Splunk 8 yet.
oh thanks Jconger. hey one more help from you, as i had mentioned in my previous chat that I have the Azure data in splunk but what kind of reports or visualization I can get out of this data in splunk, I could see Microsoft Azure App Template in splunk base but it is for Splunk Add-on for Microsoft Cloud Services. Is there any other app to play around the visualization/report.
thanks in advance.
Hi Jason Conger, Good Morning once again I have re-configured,
EventHubNamesapce, EventHub,Poilcy name, Client secret key, and all other required prerequisites mentioned in both Splunk & Microsoft Azure documents. But still I am unable to see events in Splunk.
We are getting this information for the **Eventhub and Metric inputs** in Splunk.
index="_internal" sourcetype="ta:ms:aad:log"
2020-03-16 13:18:46,617 INFO pid=25717 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2020-03-16 13:18:42,702 INFO pid=25640 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
**For Active directory audit inputs:**
2020-03-16 13:18:44,734 ERROR pid=25640 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_ms_aad/modinput_wrapper/base_modinput.py", line 127, in stream_events
self.collect_events(ew)
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_audit.py", line 84, in collect_events
input_module.collect_events(self, ew)
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/input_module_MS_AAD_audit.py", line 74, in collect_events
audit_events = azutils.get_items(helper, access_token, url, items=[])
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_azure_utils/utils.py", line 33, in get_items
raise e
HTTPError: 401 Client Error: Unauthorized for url: https://graph.microsoft.com/beta/auditLogs/directoryAudits?$orderby=activityDateTime&$filter=activit...
Referred following links to configure the prerequisites in Azure side and splunk
Kindly guide me where is the issue, as I had configured exactly the steps mentioned in the documents.
thanks in advance.
The 401 Client error you are seeing is for the Azure AD audit input instead of the Event Hub input. It is stating that the Azure AD application registration used to make the connection from Splunk to Azure does not have the appropriate permissions. Refer to this spreadsheet for the necessary input permissions needed in Azure -> http://bit.ly/Splunk_Azure_Permissions
Hi jconger, I have given the below required permission in Azure portal for Eventhub and for the Audit logs.
Azure side:
Home --> EventHubs --> Eventhubname --> shared access policies --> SAS Policy: RootManageSharedAccesskey --> claims --> Manage,send,Listen
Home --> Subscription --> Free Trial | Access control(IAM) --> Role assignments
Network Contributor, Reader, Security Admin, Security Reader
Splunk Side configuration details:
Eventhub(Preview) Inputs:
EventHub name: HXXXXXXXX
Connection String --> Endpoint=sb://Hxxxxxxx-01.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=xxxxxxxxxxxxxxxxxxxxxxxxxx=
Metric(Preview)Inputs:
Azure App Account : Azure_Mon
Tenant ID:axxxx0c-f2xxx-47b8-9xxx3-75xxx708xxx4
Subscription ID: xxxxxc1a-xxxx-4ac1-8xxx4-f7fxxxxxxx90
Microsoft Azure Active Directory Audit:
Azure App Account : Azure_Mon
Tenant ID:axxxx0c-f2xxx-47b8-9xxx3-75xxx708xxx4
After configuring the above inputs and permission, still no luck i am seeing this below error in splunk internal logs
index="_internal" sourcetype="ta:ms:aad:log" connectionpool.py:_new_conn:758
2020-03-19 18:03:48,706 INFO pid=9135 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
index="_internal" sourcetype="ta:ms:aad:log" file="setup_util.py:log_info:114"
2020-03-19 18:49:49,320 INFO pid=29789 tid=MainThread file=setup_util.py:log_info:114 | Proxy is not enabled!
index="_internal" sourcetype="ta:ms:aad:log" file="setup_util.py:log_info:114 | Log level is not set"
2020-03-19 18:05:25,956 INFO pid=9953 tid=MainThread file=setup_util.py:log_info:114 | Log level is not set, use default INFO
index="_internal" sourcetype="ta:ms:aad:log" file="client_abstract.py:__init__:161"
2020-03-19 18:50:59,195 INFO pid=30328 tid=MainThread file=client_abstract.py:__init__:161 | u'eventhub.pysdk-07a5fdec': Created the Event Hub client
index="_internal" sourcetype="ta:ms:aad:log" file="splunk_rest_client.py:_request_handler:100"
2020-03-19 18:50:45,261 INFO pid=30224 tid=MainThread file=splunk_rest_client.py:_request_handler:100 | Use HTTP connection pooling
index="_internal" sourcetype="ta:ms:aad:log" file="base_modinput.py:log_error:307"
2020-03-19 18:53:14,176 ERROR pid=31612 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_ms_aad/modinput_wrapper/base_modinput.py", line 127, in stream_events
self.collect_events(ew)
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_audit.py", line 84, in collect_events
input_module.collect_events(self, ew)
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/input_module_MS_AAD_audit.py", line 74, in collect_events
audit_events = azutils.get_items(helper, access_token, url, items=[])
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_azure_utils/utils.py", line 33, in get_items
raise e
HTTPError: 401 Client Error: Unauthorized for url: https://graph.microsoft.com/beta/auditLogs/directoryAudits?$orderby=activityDateTime&$filter=activit...
Kindly guide me how to fix this issue, we are finding it difficult to fix this issue.
Here is a screenshot of what the permissions on the application should be. Also, be sure to click the "Grant admin consent..." button.
Hi jconger thanks for sharing the screen shot, I would like to share my steps with the screen shot, So is there any other way I can share the details with you apart from answers.com as I am not sure how to attach the screen in the answers.com.
You should be able to post screen shots by uploading them somewhere publicly accessible and then using Markdown to include them in your post. Something like this:
![alt text][1]
[1]: https://cdn.apps.splunk.com/media/public/docimages/6245b8de-6a2e-11ea-ab4f-0a356b057c84.png
But, feel free to send me an email also - jconger (at) splunk (dot) com
Hi Jconger, Good Morning, I had sent you the email with the steps for both Azure and Splunk configuration, Kindly do guide me to fix the issue.
Hi Jconger, Good Morning, I am still facing an issue in ingesting the Azure Event Hub details in splunk. Regarding this I had sent you mail with the configuration steps and error details screen shots. Kindly guide me how to troubleshoot and fix the issue.
Hi Jconger, I was able to ingest most of the Microsoft azure inputs into splunk, thanks for guiding to resolve the issue.
But still I am not sure what kind of out put we will be able to see when we ingest Azure Event Hub inputs in Splunk. As I am unable to see any output for this input alone.
Now I have the Azure data in splunk but what kind of visualization I can get out of this data in splunk, I could see Microsoft Azure App Template in splunk base but it is for Splunk Add-on for Microsoft Cloud Services. So is there any other way I can get some report/dashboard using Microsoft Azure add-on for splunk.
I am getting same error not sure where is the problem so could anyone guide me in resolving this issue.
index="_internal" sourcetype="ta:ms:aad:log" source="/opt/splunk/var/log/splunk/ta_ms_aad_MS_AAD_audit.log"
2020-03-16 17:40:26,976 ERROR pid=5422 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_ms_aad/modinput_wrapper/base_modinput.py", line 127, in stream_events
self.collect_events(ew)
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_audit.py", line 84, in collect_events
input_module.collect_events(self, ew)
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/input_module_MS_AAD_audit.py", line 74, in collect_events
audit_events = azutils.get_items(helper, access_token, url, items=[])
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_azure_utils/utils.py", line 33, in get_items
raise e
HTTPError: 401 Client Error: Unauthorized for url: https://graph.microsoft.com/beta/auditLogs/directoryAudits?$orderby=activityDateTime&$filter=activit...
index="_internal" sourcetype="ta:ms:aad:log" source="/opt/splunk/var/log/splunk/ta_ms_aad_azure_event_hub.log"
2020-03-16 17:40:25,196 INFO pid=5444 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
index="_internal" sourcetype="ta:ms:aad:log" source="/opt/splunk/var/log/splunk/ta_ms_aad_azure_metrics.log"
2020-03-16 17:46:15,387 INFO pid=8584 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1