TA-MS-AAD Signin logs missing - help!

[KendallW]

Hey gang, 

I'm using the Splunk Add on for Microsoft Azure to ingest AAD signin logs to Splunk under the azure:aad:signin sourcetype, however there seems to be a gap between the number of events visible in EntraID versus what is visible from Splunk. There are always slightly more events in EntraID. The gap seems to worsen the higher the volume of events becomes. See this table:

Time	Splunk	Entra ID	Difference
1st hour	3265	3305	40
2nd hour	3085	4804	1719
3rd hour	3264	6309	3045
4th hour	2274	3841	1567
5th hour	1659	2632	973
6th hour	2168	3442	1274
7th hour	6236	8923	2687
8th hour	22716	35901	13185
9th hour	63186	101602	38416
10th hour	88607	145503	56896
11th hour	68407	140095	71688
12th hour	76866	124423	47557
13th hour	68717	122355	53638
14th hour	81310	144880	63570
15th hour	50849	140876	90027
16th hour	42972	124040	81068
17th hour	33693	91792	58099
18th hour	13683	50408	36725
19th hour	13973	38695	24722
20th hour	12182	29645	17463
21st hour	9734	24187	14453
22nd hour	8037	16935	8898
23rd hour	5869	11994	6125
24th hour	5631	8837	3206
Total	688383	1385424	697041
Percentage difference			50.31%

- This gap appears even when searching historical logs i.e. time slots over the last two weeks.
- The retention period of the index is 90 days, so the events should not have expired yet.
- There are no line breaking, event breaking, aggregation, timestamp, or other parsing errors for the sourcetype. - The gap is still present when searching over all time.
- The internal logs from the Splunk Add on for Microsoft Azure only show the following two error messages which don't seem relevant, and only appeared a few times over the last month or so:
"File "/opt/splunk/etc/apps/TA-MS-AAD/lib/splunklib/binding.py", line 1337, in request
raise HTTPError(response)
splunklib.binding.HTTPError: HTTP 503 Service Unavailable -- KV Store is in maintenance mode."
"File "/opt/splunk/etc/apps/TA-MS-AAD/lib/splunklib/modularinput/event.py", line 111, in write_to
stream.flush()
BrokenPipeError: [Errno 32] Broken pipe"

I have updated the polling interval of the Microsoft Entra ID Interactive Sign-ins input to 900 seconds, but still the issue persists.

What other explanation could there be for the gap?

Thanks,
K

[isoutamo]

Hi
if/when this is Splunk supported TA then just create a support case.
I supposing that there is some issue to reading those events from EntraId. After it has tried to read it but for unknown reason it has failed with some, it will write check point (describe what it has read). Then it start with that check point on next round and miss some events which has incorrectly marked as read. Or something similar. Any how inform the creator of TA.

[KendallW]

Hi @isoutamo thanks for your reply,

Unfortunately the TA is not supported by Splunk, and Splunk support told me as much when I raised a case with them and suggested I try the forum. 

I did find this post which seems to be a very similar issue, so we are going to try the solution there of using the Splunk Add-on for Microsoft Cloud Services (which IS Splunk supported!) to ingest the events with HEC via Event Hub. 

I will reply again with any updates