Splunk Add on for Microsoft Azure 404 Client Error: Not Found for url- How would I fix?

[matthewroberson]

I am trying to configure the Splunk Add on for Microsoft Azure (version 4.0.2 on a stand alone Heavy Forwarder running version 9.0.1 of splunk, os RHEL 7) and I'm seeing the error below in /opt/splunk/var/log/splunk/ta_ms_aad_MS_AAD_audit.log.

2022-09-14 11:41:41,871 ERROR pid=12784 tid=MainThread file=base_modinput.py:log_error:316 | Get error when collecting events.
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/TA-MS-AAD/lib/splunktaucclib/modinput_wrapper/base_modinput.py", line 140, in stream_events
    self.collect_events(ew)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_audit.py", line 168, in collect_events
    response = azutils.get_items_batch_session(helper=helper, url=url, session=session)
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_azure_utils/utils.py", line 119, in get_items_batch_session
    raise e
  File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_azure_utils/utils.py", line 115, in get_items_batch_session
    r.raise_for_status()
  File "/opt/splunk/etc/apps/TA-MS-AAD/lib/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://graph.microsoft.com/None/auditLogs/directoryAudits?$orderby=activityDateTime&$filter=activityDateTime+gt+2021-10-01T14:26:12.017133Z+and+activityDateTime+le+2022-09-14T16:34:41.623739Z

On the Azure (Government) side we have the permissions below enabled:
AuditLog.Read.All
Device.Read.All
Directory.Read.All
Group.Read.All
GroupMember.ReadWrite.All
IdentityRiskEvent.Read.All
Policy.Read.All
Policy.Read.ConditionalAccess
Policy.ReadWrite.ConditionalAccess
SecurityEvents.Read.All
User.Read
User.Read.All

Also, we have a P2 license so that should not be the issue. We have a python script that is able to retrieve signins from Azure using the same credentials we are using for the Splunk Add on for Microsoft Azure.

Another thing I noticed is the url in the error message seem wrong. Seems like it should be:

https://graph.microsoft.com/v1.0/auditLogs/directoryAudits$orderby=activityDateTime&$filter=activityDateTime+gt+2021-10-01T14:26:12.017133Z+and+activityDateTime+le+2022-09-14T16:34:41.623739Z

A couple of other tidbits. The app works for our commercial tenant. Our government tenant is new and at this point doesn't have any subscriptions. Does anyone know if having more than zero subscriptions is a requirement for this app?

[mkhasan]

I am facing the same problem. I have configured via web UI on the Splunk cloud IDM instance. I am getting the aad device, sign-in, groups and identity protection from the same MS Azure graph API with the tenant ID and app account. In my case, this is appearing as v1.0 but getting error HTTP error for the audit logs.

2022-09-26 23:45:24,180 ERROR pid=25942 tid=MainThread file=base_modinput.py:log_error:316 | Get error when collecting events. Traceback (most recent call last): File "/opt/splunk/etc/apps/TA-MS-AAD/lib/splunktaucclib/modinput_wrapper/base_modinput.py", line 140, in stream_events self.collect_events(ew) File "/opt/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_audit.py", line 168, in collect_events response = azutils.get_items_batch_session(helper=helper, url=url, session=session) File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_azure_utils/utils.py", line 119, in get_items_batch_session raise e File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_azure_utils/utils.py", line 115, in get_items_batch_session r.raise_for_status() File "/opt/splunk/etc/apps/TA-MS-AAD/lib/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://graph.microsoft.com/v1.0/auditLogs/directoryAudits?$orderby=activityDateTime&$filter=activit...

Microsoft Graph Permission:

AuditLog.Read.All
Device.Read.All
Directory.Read.All
Group.Read.All
IdentityRiskEvent.Read.All
User.Read
User.Read.All
Input configuration:
Name = aad_audit
Audit Sourcetype = azure:aad:audit
azure_app_account = <my config account>
endpoint = v1.0
environment = public
index = azure
interval = 86400
query_backoff_throttle = 420
query_window_size = 0
tenant_id = <my tenant ID>

[jconger]

In your case, it looks like the start date is too far in the past (note the $filter parameter of 2020-03-24)

https://graph.microsoft.com/v1.0/auditLogs/directoryAudits?$orderby=activityDateTime&$filter=activityDateTime+gt+2020-03-24T15:21:02.3519945Z+and+activityDateTime+le+2022-09-26T23:38:24.006432Z

I don't see a start date in your inputs.conf parameters posted, so it could be using an old checkpoint date for the start date.  The checkpoint is stored in the KV store, and you can clear it out with a combination of | inputlookup and | outputlookup, but it is easier to just create a new input with a different name since checkpoints key off of the input name.

[mkhasan]

Hi @jconger , Thanks for your advice. I have created a new input named Prod_AAD_Audit5 and can see the start time is correct by going back to the past seven days. "activityDateTime+gt+2022-09-21T23:56:27.306392Z"

Unfortunately getting the connection was broken by 'ReadTimeoutError" and finally stopped by the "Max retries exceeded with URL" after five times retrying.

"2022-09-28 23:56:53,529 ERROR pid=25364 tid=MainThread file=base_modinput.py:log_error:316 | Get error when collecting events.
Traceback (most recent call last):
File ""/opt/splunk/etc/apps/TA-MS-AAD/lib/urllib3/connectionpool.py"", line 449, in _make_request
six.raise_from(e, None) 

File ""/opt/splunk/etc/apps/TA-MS-AAD/lib/urllib3/util/retry.py"", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='graph.microsoft.com', port=443): Max retries exceeded with url: /v1.0/auditLogs/directoryAudits?$orderby=activityDateTime&$filter=activityDateTime+gt+2022-09-21T23:56:27.306392Z+and+activityDateTime+le+2022-09-28T23:49:27.438651Z (Caused by ReadTimeoutError(""HTTPSConnectionPool(host='graph.microsoft.com', port=443): Read timed out. (read timeout=5)"")) 

2022-09-28 23:56:27,438 DEBUG pid=25364 tid=MainThread file=base_modinput.py:log_debug:298 | _Splunk_ input_name=Prod_AAD_Audit5 Audit URL used: https://graph.microsoft.com/v1.0/auditLogs/directoryAudits?$orderby=activityDateTime&$filter=activityDateTime+gt+2022-09-21T23:56:27.306392Z+and+activityDateTime+le+2022-09-28T23:49:27.438651Z

2022-09-28 23:56:32,458 DEBUG pid=25364 tid=MainThread file=retry.py:increment:594 | Incremented Retry for (url='/v1.0/auditLogs/directoryAudits?$orderby=activityDateTime&$filter=activityDateTime+gt+2022-09-21T23:56:27.306392Z+and+activityDateTime+le+2022-09-28T23:49:27.438651Z'): Retry(total=2, connect=3, read=2, redirect=None, status=None)

[jconger]

Are you creating inputs via the UI or editing inputs.conf directly? The endpoint used (the part between graph.microsoft.com/ and /auditLogs ) comes from the endpoint parameter in inputs.conf

[MS_AAD_audit://aad_audit]
audit_sourcetype = azure:aad:audit
azure_app_account = test
endpoint = v1.0
environment = public
index = azure
interval = 86400
query_backoff_throttle = 420
query_window_size = 0
tenant_id = REDACTED

[matthewroberson]

I'm creating inputs through the GUI. And it turned out to be an input for the commercial tenant ( not the government tenant ) that was throwing the error.

Should the signin input for the government tenant work if we haven't created a subscription yet in the government tenant?

We have a python script that is able to connect and pull back signins, but the input we created in the app isn't working. To help with troubleshooting, I disabled all the inputs except the signins input for the government tenant. What I see in the logs every time the input is scheduled to check is:

2022-09-27 12:08:11,617 DEBUG pid=31970 tid=MainThread file=connectionpool.py:_make_request:465 | https://127.0.0.1:8089 "GET /servicesNS/nobody/TA-MS-AAD/storage/collections/data/TA_MS_AAD_checkpointer/aad_signin_last_date_ADSignins_SPP_GCC HTTP/1.1" 404 140
127.0.0.1 - splunk-system-user [27/Sep/2022:12:08:11.615 -0500] "GET /servicesNS/nobody/TA-MS-AAD/storage/collections/data/TA_MS_AAD_checkpointer/aad_signin_last_date_ADSignins_SPP_GCC HTTP/1.1" 404 140 "-" "curl" - - - 1ms
2022-09-27 12:08:11,614 DEBUG pid=31970 tid=MainThread file=binding.py:get:695 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-MS-AAD/storage/collections/data/TA_MS_AAD_checkpointer/aad_signin_last_date_ADSignins_SPP_GCC (body: {})
2022-09-27 12:08:11,613 DEBUG pid=31970 tid=MainThread file=connectionpool.py:_make_request:465 | https://127.0.0.1:8089 "GET /servicesNS/nobody/TA-MS-AAD/storage/collections/config/?count=-1&offset=0&search=TA_MS_AAD_checkpointer HTTP/1.1" 200 4505
127.0.0.1 - splunk-system-user [27/Sep/2022:12:08:11.611 -0500] "GET /servicesNS/nobody/TA-MS-AAD/storage/collections/config/?count=-1&offset=0&search=TA_MS_AAD_checkpointer HTTP/1.1" 200 4505 "-" "curl" - - - 1ms
2022-09-27 12:08:11,610 DEBUG pid=31970 tid=MainThread file=binding.py:get:695 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-MS-AAD/storage/collections/config/ (body: {'count': -1, 'offset': 0, 'search': 'TA_MS_AAD_checkpointer'})
2022-09-27 12:08:11,609 DEBUG pid=31970 tid=MainThread file=connectionpool.py:_make_request:465 | https://127.0.0.1:8089 "GET /servicesNS/nobody/TA-MS-AAD/storage/collections/config/TA_MS_AAD_checkpointer HTTP/1.1" 200 5353
127.0.0.1 - splunk-system-user [27/Sep/2022:12:08:11.608 -0500] "GET /servicesNS/nobody/TA-MS-AAD/storage/collections/config/TA_MS_AAD_checkpointer HTTP/1.1" 200 5353 "-" "curl" - - - 1ms
2022-09-27 12:08:11,598 DEBUG pid=31970 tid=MainThread file=binding.py:get:695 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-MS-AAD/storage/collections/config/TA_MS_AAD_checkpointer (body: {})
127.0.0.1 - splunk-system-user [27/Sep/2022:12:08:11.587 -0500] "GET /servicesNS/nobody/TA-MS-AAD/configs/conf-ta_ms_aad_settings/logging?output_mode=json HTTP/1.1" 200 1534 "-" "curl" - - - 1ms
127.0.0.1 - splunk-system-user [27/Sep/2022:12:08:11.553 -0500] "GET /servicesNS/nobody/TA-MS-AAD/configs/conf-ta_ms_aad_settings/_reload HTTP/1.1" 200 2025 "-" "curl" - - - 22ms
127.0.0.1 - splunk-system-user [27/Sep/2022:12:08:11.539 -0500] "GET /servicesNS/nobody/TA-MS-AAD/TA_MS_AAD_settings/logging?output_mode=json&count=0&--cred--=1 HTTP/1.1" 200 1071 "-" "curl" - - - 55ms
127.0.0.1 - splunk-system-user [27/Sep/2022:12:08:11.517 -0500] "GET /servicesNS/nobody/TA-MS-AAD/storage/passwords?count=-1&offset=0 HTTP/1.1" 200 41679 "-" "curl" - - - 2ms
127.0.0.1 - splunk-system-user [27/Sep/2022:12:08:11.504 -0500] "GET /servicesNS/nobody/TA-MS-AAD/configs/conf-ta_ms_aad_settings/proxy?output_mode=json HTTP/1.1" 200 1501 "-" "curl" - - - 1ms
127.0.0.1 - splunk-system-user [27/Sep/2022:12:08:11.470 -0500] "GET /servicesNS/nobody/TA-MS-AAD/configs/conf-ta_ms_aad_settings/_reload HTTP/1.1" 200 2025 "-" "curl" - - - 23ms
127.0.0.1 - splunk-system-user [27/Sep/2022:12:08:11.457 -0500] "GET /servicesNS/nobody/TA-MS-AAD/TA_MS_AAD_settings/proxy?output_mode=json&count=0&--cred--=1 HTTP/1.1" 200 1106 "-" "curl" - - - 70ms
127.0.0.1 - splunk-system-user [27/Sep/2022:12:08:11.404 -0500] "GET /servicesNS/nobody/TA-MS-AAD/data/inputs/MS_AAD_signins?output_mode=json&count=0 HTTP/1.1" 200 4606 "-" "curl" - - - 33ms
127.0.0.1 - splunk-system-user [27/Sep/2022:12:08:11.391 -0500] "GET /servicesNS/nobody/TA-MS-AAD/TA_MS_AAD_MS_AAD_signins?output_mode=json&count=0&--cred--=1 HTTP/1.1" 200 3872 "-" "curl" - - - 50ms
127.0.0.1 - splunk-system-user [27/Sep/2022:12:08:11.369 -0500] "GET /servicesNS/nobody/TA-MS-AAD/storage/passwords?count=-1&offset=0 HTTP/1.1" 200 41679 "-" "curl" - - - 2ms
127.0.0.1 - splunk-system-user [27/Sep/2022:12:08:11.356 -0500] "GET /servicesNS/nobody/TA-MS-AAD/configs/conf-ta_ms_aad_account?output_mode=json&count=0 HTTP/1.1" 200 4096 "-" "curl" - - - 1ms
127.0.0.1 - splunk-system-user [27/Sep/2022:12:08:11.323 -0500] "GET /servicesNS/nobody/TA-MS-AAD/configs/conf-ta_ms_aad_account/_reload HTTP/1.1" 200 2020 "-" "curl" - - - 21ms
127.0.0.1 - splunk-system-user [27/Sep/2022:12:08:11.310 -0500] "GET /servicesNS/nobody/TA-MS-AAD/TA_MS_AAD_account?output_mode=json&count=0&--cred--=1 HTTP/1.1" 200 3028 "-" "curl" - - - 68ms