All Apps and Add-ons

Azure AD user data (azure:aad:user) is ingesting the first 100 users over and over again every second.

def65483
Explorer

I have logging set to debug. nothing interesting except that it is pulling in the exact same skip token (100 users) every second nonstop.
I have completely removed the input and made another with a new name. But, it does the exact same thing. No errors, just the same graph call every.... single... second...

1 Solution

sleclerc1
Explorer

Found a solution, was working with version 2.1.0:

This code is a snippet from "input_module_MS_AAD_user.py", located at $SPLUNK_HOME/etc/apps/TA-MS-AAD/bin

First, I found the syntax error in the second item, "nusers...". After changing that, the query was working successfully, but returning no data. I then tried the v1.0 version and success! Our organization has >600K users and the query took over an hour.

EDIT:

After additional troubleshooting I simplified the answer even more. I thought I had found two problems, but actually only one.

The second part I thought that also needed fixing was the version of the API it was using. I thought that v1.0 had to be used, and while it does work, and pulls significantly less fields than it's beta variant, and is not compatible with the Microsoft Azure App for Splunk. I thought it wasn't working because the BETA API responses were yielding an HTTP 200 w/ 0 bytes transferred....but reviewing logs in our Splunk environment, it was successfully querying/ingesting the data.

View solution in original post

0 Karma

sleclerc1
Explorer

Found a solution, was working with version 2.1.0:

This code is a snippet from "input_module_MS_AAD_user.py", located at $SPLUNK_HOME/etc/apps/TA-MS-AAD/bin

First, I found the syntax error in the second item, "nusers...". After changing that, the query was working successfully, but returning no data. I then tried the v1.0 version and success! Our organization has >600K users and the query took over an hour.

EDIT:

After additional troubleshooting I simplified the answer even more. I thought I had found two problems, but actually only one.

The second part I thought that also needed fixing was the version of the API it was using. I thought that v1.0 had to be used, and while it does work, and pulls significantly less fields than it's beta variant, and is not compatible with the Microsoft Azure App for Splunk. I thought it wasn't working because the BETA API responses were yielding an HTTP 200 w/ 0 bytes transferred....but reviewing logs in our Splunk environment, it was successfully querying/ingesting the data.

View solution in original post

0 Karma

def65483
Explorer

I did something similar by just editing the line in place to force the next page link to the one in the manifest i saved down. It's annoying, but i'm happy someone else is seeing this issue. it is very odd each page is referencing itself 😕

I've asked some MS peeps as well. I'll post an update as soon as i hear anything back.

0 Karma

sleclerc1
Explorer

To my understanding, the syntax error I found caused the API to loop the original call, rather than actually use the next page link.

0 Karma

def65483
Explorer

Also, your attachment is not showing. :(. I suspect you are referring to line: 54 and fixing so it creates the user response instead of "nuser" response.

I had a similar issue when changing it where i still had a previous shim in place and made it fail entirely but yea. thats the only part i see. big oof there.

0 Karma

sleclerc1
Explorer

Yeah looks like the image was uploaded to a "temp" location, so I guess it was deleted. But yes, you are correct.

Nothing like a single character syntax error to break the entire API haha

0 Karma

def65483
Explorer

ahhh i see that now. lol whyyyyyy

0 Karma

def65483
Explorer

I'll note that the skip token pull and the @odata.nextlink are always the same (itself).

And now that i look at it closer its more like 2-3 times per second (depending on how fast i can pull 100 users)

0 Karma

sleclerc1
Explorer

Same issue. Been trying to dig in to odata.nextlink and pagination, but haven't turned up anything useful to assist. I've event duplicated the issue within Graph Explorer; performing the user pull only grabs the first 100 users, but I see the pagination "attempt". I have a feeling it's on Microsoft's side... Although I hope I'm wrong.

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!