SHC deployer, not deleting TA's (troubleshooting Or 9.4.0 bug?)

[NullZero]

IHAC with an SVA C3 (On-Prem) setup running 9.4.0 on the MN, SHC, Deployer but 9.3.2 on the peers (upgrade in the works due to unsupported linux kernel 3.x). They've been running this way OK for about a month whilst the upgrade is pending.

Start of issue
The problem that is being seen is that the client wanted to disable the new 'audit_trail' app for platform confidentiality a week ago. They created a local folder for the app on the deployer ($SPLUNK_HOME/etc/shcluster/apps/audit_trail) and disabled it via a .conf file change, no issue worked ok and pushed to the SHC from the deployer. The SHC is all in sync.

Symptom
The issue now being seen is that they can't delete TA's and apps with pushes from the Deployer. For example they are removing legacy TA's and despite not being on the deployer they remain on the SHC. The cluster is operational and in sync OK and I have temporarily removed the 'audit_trail' workaround which allows the usual command to operate again:

./splunk apply shcluster-bundle -target <https://x.x.x.x:8089> -preserve-lookups true

If not you have to include the switch (-push-default-apps true)

Next steps:

• I'm trying to locate the correct component in index _internal to troubleshoot what is happening and why it is not deleting apps and TA's not on the Deployer
• Example:
• index="_internal" source="/opt/splunk/var/log/splunkd.log" host IN (SH, SH, SH, Deployer)
• I can't locate any warnings or relevant errors even when including the relevant TA being intended for removal on the short time period in question
• Any suggestions welcome

[NullZero]

Thanks for trying to help @PickleRick , this is of course the normal process that I have undertaken and iterated over to analyse. Understood on disabled, that is not the case in the TA's I am working to remove.

[PickleRick]

OK. So the TA you're trying to remove is disabled or enabled?

[NullZero · Accepted solution]

Correct, I made sure it was not was NOT disabled as a process of elimination in the troubleshooting.

Resolution:

• Having made sure it was not on the deployer Or in '/opt/splunk/var/run/splunk/deploy/apps/' I manually deleted the TA folder and undertook a rolling restart on the SHC. This fixed it.
• Prior to this I had also found WARN in _internal relating to deprecated parameters in limits.conf, planning a change tomorrow to support the updated stanza / autorize params.
  [auth]
  enable_install_apps = true
• I also noted that in the given app under app.conf there was a niche setting:
  allows_disable = false
  I'm unclear if this has any impact on deletion (docs don't say).

[PickleRick]

https://docs.splunk.com/Documentation/Splunk/latest/DistSearch/PropagateSHCconfigurationchanges

"To delete an app that you previously pushed, remove it from the configuration bundle. When you next push the bundle, each member will delete it from its own file system. Note: If you need to remove an app, inspect its app.conf file to make sure that state = enabled. If state = disabled, the deployer will not remove the app even if you remove it from the configuration bundle."