topic Re: How to tell when it's safe to query the API about a newly created user? in Getting Data In

How to tell when it's safe to query the API about a newly created user?

krisreeves — Sun, 03 May 2020 15:05:45 GMT

When using SSO with clustered search heads, users who lose SSO access leave behind knowledge objects and directories on the file system. I'm doing some work to clean these up. In order to be able to query the Splunk API for the full set of information, it's necessary to re-create the user so that Splunk will see and return information about their private knowledge objects. While doing this, I noticed the following problem:

I can ask the API about the knowledge objects owned by a user via /servicesNS/-/-/admin/directory
However, if I ask about saved searches from /servicesNS/USERNAME/search/saved/searches/SEARCH_NAME, I get a 404 for searches that are present in (1)
After 5-10 seconds, (2) begins to work

I suspected that maybe the search head cluster needs to sync some configuration, so I hit /services/shcluster/status while (2) is failing repeatedly to get some info about the search head cluster state. None of the search heads specify they are out of sync, and the last conf replication time reset (indicating a configuration replication had happened), but the API was still returning 404s on saved searches for a few seconds.

Is there any way to know when it's "safe" to request information pertaining to a user? Is the /directory endpoint potentially affected by this? Are there other endpoints that may be affected in the same way?

One other thing I tested was querying /servicesNS/-/search/saved/searches/SEARCH_NAME, however it exhibited the same behavior. Not all users seem to behave this way, but the particular user in question had a couple knowledge objects of type "props-extract". It seems likely that re-adding those to the system is taking longer, and this added delay somehow affects their saved searches showing up.

Re: How to tell when it's safe to query the API about a newly created user?

jkat54 — Sun, 03 May 2020 22:03:03 GMT

You should be able to hit /debug/refresh first then the endpoint you're looking for. In SHC, if you're aiming directly at one server versus querying a load balancer VIP you shouldn't have to worry about propagation.

Re: How to tell when it's safe to query the API about a newly created user?

krisreeves — Mon, 04 May 2020 06:22:38 GMT

All my testing was against a single host (no load balancer involved) from localhost, which is why I'm puzzled. By propagation, I was thinking that perhaps once the user is enabled, some system process needed to atomically update the cluster (due to the props-extract configs, which none of the other users that seem to work fine have) before it would "activate", much like cluster bundle config pushes, and that I was hitting things before that happened.

I'll see if /debug/refresh makes a difference tomorrow, since the problem seems to be consistently reproducible with this one particular user.

Re: How to tell when it's safe to query the API about a newly created user?

krisreeves — Mon, 04 May 2020 16:26:31 GMT

(update)

/debug/refresh is a web ui thing, not a REST API thing. I found the comment below this answer: https://answers.splunk.com/answering/661816/view.html and gave it a try, but the same behavior is still exhibited (I have to retry multiple times to get the expected data)

Re: How to tell when it's safe to query the API about a newly created user?

jkat54 — Mon, 04 May 2020 17:13:00 GMT

https://answers.splunk.com/answers/565493/is-there-a-cli-command-for-debug-refresh.html

From above... try this:

curl -u admin -X POST https://localhost:8089/servicesNS/-/-/admin/savedsearch/_reload

Re: How to tell when it's safe to query the API about a newly created user?

krisreeves — Mon, 04 May 2020 19:10:14 GMT

That's exactly what I did. "I found the comment below this answer and gave it a try"
It didn't help 😞