There are 2 issues as per the logs you have mentioned here, 1. ERROR related to the tablespace which usually reffers to the issue with the database and if you are using DB connect or Internal KV store.  2. CredentialNotExistException: This is with regards to the Credentials and also splunk is unable to fetch the stored credentials from the passwords.conf. Strongly feel that there is an corrupted entry in the KV store, and also let me know if there was an app which was restored and the credentials are the same too.  Check in the logs on which app is failing because of the passwords and also check for the kvstore status and try to restart the splunk. If the Splunk doesnt get restarted then try to clean the kv store and then try to restart the splunk.   ... View more

The straight forward answer to your question is "NO".  Splunk 9.x, particularly the Universal Forwarder, has introduced changes related to security and user management. This includes the introduction of a new "least privileged" splunkfwd user for managing the forwarder on Linux, and potentially stricter requirements for TTY allocation during startup in certain scenarios.  When running in a containerized environment like Kubernetes, the lack of a TTY or specific user permissions can lead to the process hanging as it expects an interactive session or fails to perform actions without the necessary privileges. ... View more

I would like to say that its always better to fix the problem at the source and try to mask those details which ever is not needed. If you are not able to then you should have a regex in place and try to check that these raw events match those regex and then try to mask it.  One more way is to have a program running, where you can have the raw events having matching the regex and convert them into a different one this way it will definitely help you. I am sure by now you would have got the answers from the team of experts but however it all depends on your organizations and how well you can perform it.  Below is a link which will help you for the masking of events.  https://help.splunk.com/en/splunk-enterprise/get-started/get-data-in/9.4/configure-event-processing/anonymize-data ... View more

Thanks  @dural_yyz,  you were right. I tried to copy the files from the Splunk manifest under the $plunk_Home and use the same in the latest version manifest file in the Splunk 9.x and after that I restarted the services and I dont see the error again. Looks like When the splunk starts it reads the manifest file and triggers the error that are not in the manifiest file.  Cheers !!!! ... View more

@kiran_panchavat , @PickleRick , @ITWhisperer , @isoutamo , @bowesmana  ... View more

@dural_yyz, @amahoski , @gcusello  ... View more