Deployment Server clients have wrong apps

[tlmayes]

We have a small satellite deployment of 40+ servers, that have a dedicated HF doubling as a Deployment Server running on Linux.  Equal mix of Windows and Linux.  24h ago discovered that a few of the Windows servers were now reporting that they no longer had the Windows_TA installed, but instead were running the Linux_TA.  Checking the UF hosts directly, they in fact were running the Windows_TA even though the DS was reporting they were running the Linux_TA??

After a day of trying to figure out how (validated filters, tested, removed and readded all Server Classes, and Apps), it continued.  Noticed throughout the day a few more were now reporting this "mix-up", and again validated those reporting Linux_TA were running Windows_TA.  As a final drastic measure, removed Splunk from the host (the HF/DS, not the UF's), reinstalled from scratch, and created the environment new.  Made sure the UF's were not running any of the distributed apps/ta's.  Built new Apps, Server Class.  The UF's started phoning home, and once again, the Windows servers were reporting the Linux_TA, but running the Windows_TA

[tlmayes]

Everything is shiny "new".  This is a satellite to our full implementation, hosted in AWS. 
Splunk 9.2.0.1 on both agents and the DS (which doubles as an HF) running on AWS RHEL 8.9.  UF's are all running 9.2.0.  Less than 40 total agents (14 Win, 26 nix). 

DS was acting up, so destroyed it and built new.  Instantly, the same problem.  Even tried adding hostnames to the filter vice using wildcard.  Same.  The odd thing.  The DS reports that Windows hosts are running the Linux TA, but when you check the Windows hosts, they are running the Windows TA as they should be

[isoutamo]

Have you local indexes on DS or are you sending logs to your real indexers? This has changes on 9.2.x and it could cause something weird.

[tlmayes]

Great point, and something I did not know beforehand.  In troubleshooting stumbled onto the documentation stating what you are pointing out, the new _ds* indexes.  So yes, the _ds* indexes are local to the DS.