I've got some users who are no longer around in my Splunk instance and I want to remove the user created objects. Is there a procedure I can follow for this task?
The intersection of LDAP and Splunk users is a challenge when it comes to user-created objects, as Splunk won't clean-out user folders or other objects if their authentication fails.
-- splunkd.log errors seen--
ERROR UserManagerPro - Failed to get LDAP user="my_user" from any configured servers
ERROR AuthenticationManagerLDAP - Could not find user="my_user" with strategy="LDAP_or_AD_config"
ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/search/splunk/etc/users/$userid/user-prefs/metadata/local.meta: No such file or directory
-- end splunkd.log example --
Please note that with the release of Splunk 6.6, there's a feature to find, alert, and manage orphaned knowledge objects. The details are available in the documentation here. As an admin, you'd receive a daily alert, have a dashboard to review the objects, and an opportunity to reassign them to a different user. Reviewing the steps above, the "Orphaned Scheduled Searches, Reports, and Alerts" dashboard and UI to manage objects would replace Steps 2, 3, and 4 for most use-cases. Enjoy!
If you see many errors about missing user in the splunkd.log, this is because deleted LDAP users still own objects in splunk, by example a scheduled search.
and you should clean it
Delete the objects/profile or migrate them to another user or an app. See answer below.
The intersection of LDAP and Splunk users is a challenge when it comes to user-created objects, as Splunk won't clean-out user folders or other objects if their authentication fails.
-- splunkd.log errors seen--
ERROR UserManagerPro - Failed to get LDAP user="my_user" from any configured servers
ERROR AuthenticationManagerLDAP - Could not find user="my_user" with strategy="LDAP_or_AD_config"
ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/search/splunk/etc/users/$userid/user-prefs/metadata/local.meta: No such file or directory
-- end splunkd.log example --
What about Splunk cloud users? Is this something that cloud ops needs to handle or can these items be migrated through the UI?
Seeing as you don't have access to the configuration files as part of the Splunk Cloud SaaS solution - then it is 100% something Splunk should take care of. They SHOULD have alerting in place that notifies them when the error message comes up in the log entries, then resolve it - but I wouldn't be surprised if you also have to raise an incident to formally ask them to remove the user dir.
For those of you on *Nix machines would be able to do something like this:
cd $SPLUNK_HOME/etc
for x in `find . -name 'local.meta'`; do cp $x $x.old ; sed s/olduser/newuser/ < $x > $x.new ; mv -f $x.new $x ; done
This will make a backup of the local.meta, swap out the olduser for the newuser and copy it over the local.meta. All you should have to do is restart Splunk.
Brian
I wrapped this up in a script and it works perfectly - really nice solution to the problem here, bit surprised Splunk haven't got something out of the box for you to fix this problem. We had a set of administrators leave and this caused some pain.
The only addition I would make. is automating the removal of the user id folder as explained above (after creating a backup) then perform a refresh (as mentioned in https://answers.splunk.com/answers/168898/how-can-we-find-all-the-searches-alerts-dashboard.html) for the config settings to take affect - I didn't need to restart the splunk services for this to work.
Warning: Make sure you test this in development environments first, has the chance to be very costly when overwriting so many config files in bulk, even if temporarily until it can be reverted.
This is money