IHAC with an SVA C3 (On-Prem) setup running 9.4.0 on the MN, SHC, Deployer but 9.3.2 on the peers (upgrade in the works due to unsupported linux kernel 3.x). They've been running this way OK for about a month whilst the upgrade is pending. Start of issue The problem that is being seen is that the client wanted to disable the new 'audit_trail' app for platform confidentiality a week ago. They created a local folder for the app on the deployer ($SPLUNK_HOME/etc/shcluster/apps/audit_trail) and disabled it via a .conf file change, no issue worked ok and pushed to the SHC from the deployer. The SHC is all in sync. Symptom The issue now being seen is that they can't delete TA's and apps with pushes from the Deployer. For example they are removing legacy TA's and despite not being on the deployer they remain on the SHC. The cluster is operational and in sync OK and I have temporarily removed the 'audit_trail' workaround which allows the usual command to operate again: ./splunk apply shcluster-bundle -target <https://x.x.x.x:8089> -preserve-lookups true If not you have to include the switch (-push-default-apps true) Next steps: I'm trying to locate the correct component in index _internal to troubleshoot what is happening and why it is not deleting apps and TA's not on the Deployer Example: index="_internal" source="/opt/splunk/var/log/splunkd.log" host IN (SH, SH, SH, Deployer) I can't locate any warnings or relevant errors even when including the relevant TA being intended for removal on the short time period in question Any suggestions welcome
... View more