Why does Splunk continuously attempt to find a user in LDAP after the user has been removed from Active Directory?

[RJ_Grayson]

I had a Splunk user that was recently removed from Active Directory and ever since, Splunk continuously attempts to find the user in the LDAP strategy. I see the following error messages every minute or so in splunkd.log.

ERROR AuthenticationManagerLDAP - Could not find user="user" with strategy="Strategy1"

ERROR AuthenticationManagerLDAP - Could not find user="user" with strategy="Strategy2"

ERROR UserManagerPro - Failed to get LDAP user="user" from any configured servers

This user owned a lot of public and private knowledge objects/saved searches. Before the user was removed from AD, I was able to change the ownership of all the user's public objects by SEDing "owner = user" to "owner = otheruser" in all of the Splunk app's local.meta files. This worked perfectly and all publicly owned objects transferred ownership without any issues.

I've also disabled all of user's privately owned searches and knowledge objects. I even went as far as SEDing each disabled searches enableSched stanza to reflect "= 0".

It would appear that Splunk is attempting to utilize some kind of knowledge object that the user owned, but I can't seem to find any other public objects owned by this user. I've grepped all of $SPLUNK_HOME/etc/apps for the username but have come up empty. Searching all of the Splunk _internal logs before and after the timestamps of the failed LDAP binds yields no clues either.

The only thing I haven't done yet is delete the user directory from $SPLUNK_HOME/etc/users.

Does anyone know of any other ways to identify potential public objects owned by this user? Or is there a way to find which objects might have been dependent on that owner's existence?

[Jeremiah · Accepted solution]

Our process is to change ownership on all shared knowledge objects by replacing the username with the new owner's username in the metadata files. Then we backup their home directory by moving it out of the $SPLUNK_HOME/etc/users directory. Finally, we restart Splunk. That seems to clear up any LDAP errors.

[RJ_Grayson]

So this ended up doing the trick. Once the user was removed from LDAP I stopped Splunk and moved the users directory out of $SPLUNK_HOME/etc/users/ into a backup directory. After restarting Splunk it stopped complaining about the user not existing in either LDAP strategy.

I do find this behavior very strange though. I have a number of users that no longer exist in LDAP but their user directories still exist in $SPLUNK_HOME/etc/users/. When I start Splunk I receive a single error message per missing LDAP user but then the error messages stop. In the case with this one particular user, even after I had double and triple checked all of their public Knowledge Objects and Saved Searches had been transferred to a new owner, it appeared as if Splunk was attempting to reach into that users directory for something and would give an error every 2-10 seconds about how it couldn't find the user in LDAP. I still couldn't find any clues as to what or why Splunk was exhibiting this behavior. If anyone has seen this before and could enlighten me I'd be very interested in hearing an explanation.

[Jeremiah]

It could be they had private objects that were periodically refreshed. Maybe field extractions or saved searches? Even if the objects were disabled, the files would still be touched periodically to be reloaded.