Deployment Server clients have wrong apps

[tlmayes]

We have a small satellite deployment of 40+ servers, that have a dedicated HF doubling as a Deployment Server running on Linux.  Equal mix of Windows and Linux.  24h ago discovered that a few of the Windows servers were now reporting that they no longer had the Windows_TA installed, but instead were running the Linux_TA.  Checking the UF hosts directly, they in fact were running the Windows_TA even though the DS was reporting they were running the Linux_TA??

After a day of trying to figure out how (validated filters, tested, removed and readded all Server Classes, and Apps), it continued.  Noticed throughout the day a few more were now reporting this "mix-up", and again validated those reporting Linux_TA were running Windows_TA.  As a final drastic measure, removed Splunk from the host (the HF/DS, not the UF's), reinstalled from scratch, and created the environment new.  Made sure the UF's were not running any of the distributed apps/ta's.  Built new Apps, Server Class.  The UF's started phoning home, and once again, the Windows servers were reporting the Linux_TA, but running the Windows_TA

[tlmayes]

Splunk support concluded it was an "as yet discovered software bug"

[hmallett]

I have nothing to add, except to say that I have observed the same bug, where the server classes that use machine filtering display the incorrect clients in the UI.

The bug remains in version v.9.2.1

[hmallett]

I believe that this bug is planned to be fixed in 9.2.2

[meetmshah]

Would you mind sharing the serverclass.conf file?

[tlmayes]

Pretty simple.... 

serverClass:All:app:all_outputs]
restartSplunkWeb = 0
restartSplunkd = 1
stateOnClient = enabled

[serverClass:All]
whitelist.0 = *

[serverClass:Windows:app:Splunk_TA_windows]
restartSplunkWeb = 0
restartSplunkd = 1
stateOnClient = enabled

[serverClass:Linux:app:Splunk_TA_nix]
restartSplunkWeb = 0
restartSplunkd = 1
stateOnClient = enabled

[serverClass:All:app:all_deploymentclient]
restartSplunkWeb = 0
restartSplunkd = 1
stateOnClient = enabled

[serverClass:Linux]
machineTypesFilter = linux-x86_64
whitelist.0 = *

[serverClass:Windows]
machineTypesFilter = windows-x64
whitelist.0 = *

[isoutamo]

Based on docs this should works. BUT on example part those platform selections have done on app not serverclass level. Maybe you should try that?

Btw have you configured this by gui or manually with text editor?

[tlmayes]

Such a small and straightforward environment I used GUI.  I get the sense there is a bug in 9.2.0.x

[isoutamo]

Ok, then it’s best to wait resolution for your P2 case.

[meetmshah]

Seems fairly simple / basic configurations. I would suggest raising Support case to get this troubleshot and fixed.

@isoutamo thoughts?

[tlmayes]

I opened a P2 3 days ago... still waiting.  Typical

[meetmshah]

Hello @tlmayes, How are you whitelisting the hosts? Do you just want to use this nice feature of filtering everything by the OS type? Screenshot below - 

With the above way, you can create 2 separate server classes for Windows and Linux and whitelist all the hosts.

Please accept the solution and hit Karma, if this helps!

[tlmayes]

Yes, filtering by OS.  Rebuilt the DS from scratch, set filters (using the OS filter).  All Linux servers receive the Linux TA.  All Windows Servers receive the Linux TA, and confirmed the OS filter, again 😕  

[isoutamo]

Hi
can you told the base information of your environment (OS, version, splunk version, TA versions, UF versions etc.)?
Have you update something lately etc.?
r. Ismo

[tlmayes]

Everything is shiny "new".  This is a satellite to our full implementation, hosted in AWS. 
Splunk 9.2.0.1 on both agents and the DS (which doubles as an HF) running on AWS RHEL 8.9.  UF's are all running 9.2.0.  Less than 40 total agents (14 Win, 26 nix). 

DS was acting up, so destroyed it and built new.  Instantly, the same problem.  Even tried adding hostnames to the filter vice using wildcard.  Same.  The odd thing.  The DS reports that Windows hosts are running the Linux TA, but when you check the Windows hosts, they are running the Windows TA as they should be

[isoutamo]

Have you local indexes on DS or are you sending logs to your real indexers? This has changes on 9.2.x and it could cause something weird.

[tlmayes]

Great point, and something I did not know beforehand.  In troubleshooting stumbled onto the documentation stating what you are pointing out, the new _ds* indexes.  So yes, the _ds* indexes are local to the DS.