topic Deployment Server clients have wrong apps in Splunk Enterprise

Deployment Server clients have wrong apps

tlmayes — Tue, 26 Mar 2024 19:22:20 GMT

We have a small satellite deployment of 40+ servers, that have a dedicated HF doubling as a Deployment Server running on Linux. Equal mix of Windows and Linux. 24h ago discovered that a few of the Windows servers were now reporting that they no longer had the Windows_TA installed, but instead were running the Linux_TA. Checking the UF hosts directly, they in fact were running the Windows_TA even though the DS was reporting they were running the Linux_TA??

After a day of trying to figure out how (validated filters, tested, removed and readded all Server Classes, and Apps), it continued. Noticed throughout the day a few more were now reporting this "mix-up", and again validated those reporting Linux_TA were running Windows_TA. As a final drastic measure, removed Splunk from the host (the HF/DS, not the UF's), reinstalled from scratch, and created the environment new. Made sure the UF's were not running any of the distributed apps/ta's. Built new Apps, Server Class. The UF's started phoning home, and once again, the Windows servers were reporting the Linux_TA, but running the Windows_TA

Re: Deployment Server clients have wrong apps

meetmshah — Wed, 27 Mar 2024 18:33:18 GMT

Hello @tlmayes, How are you whitelisting the hosts? Do you just want to use this nice feature of filtering everything by the OS type? Screenshot below -

With the above way, you can create 2 separate server classes for Windows and Linux and whitelist all the hosts.

Please accept the solution and hit Karma, if this helps!

Re: Deployment Server clients have wrong apps

tlmayes — Wed, 27 Mar 2024 18:39:49 GMT

Yes, filtering by OS. Rebuilt the DS from scratch, set filters (using the OS filter). All Linux servers receive the Linux TA. All Windows Servers receive the Linux TA, and confirmed the OS filter, again 😕

Re: Deployment Server clients have wrong apps

meetmshah — Wed, 27 Mar 2024 18:41:58 GMT

Would you mind sharing the serverclass.conf file?

Re: Deployment Server clients have wrong apps

tlmayes — Wed, 27 Mar 2024 19:02:24 GMT

Pretty simple....

serverClass:All:app:all_outputs] restartSplunkWeb = 0 restartSplunkd = 1 stateOnClient = enabled [serverClass:All] whitelist.0 = * [serverClass:Windows:app:Splunk_TA_windows] restartSplunkWeb = 0 restartSplunkd = 1 stateOnClient = enabled [serverClass:Linux:app:Splunk_TA_nix] restartSplunkWeb = 0 restartSplunkd = 1 stateOnClient = enabled [serverClass:All:app:all_deploymentclient] restartSplunkWeb = 0 restartSplunkd = 1 stateOnClient = enabled [serverClass:Linux] machineTypesFilter = linux-x86_64 whitelist.0 = * [serverClass:Windows] machineTypesFilter = windows-x64 whitelist.0 = *

Re: Deployment Server clients have wrong apps

isoutamo — Wed, 27 Mar 2024 19:03:31 GMT

Hi
can you told the base information of your environment (OS, version, splunk version, TA versions, UF versions etc.)?
Have you update something lately etc.?
r. Ismo

Re: Deployment Server clients have wrong apps

meetmshah — Wed, 27 Mar 2024 19:05:20 GMT

Seems fairly simple / basic configurations. I would suggest raising Support case to get this troubleshot and fixed.

@isoutamo thoughts?

Re: Deployment Server clients have wrong apps

tlmayes — Wed, 27 Mar 2024 19:30:12 GMT

Everything is shiny "new". This is a satellite to our full implementation, hosted in AWS.
Splunk 9.2.0.1 on both agents and the DS (which doubles as an HF) running on AWS RHEL 8.9. UF's are all running 9.2.0. Less than 40 total agents (14 Win, 26 nix).

DS was acting up, so destroyed it and built new. Instantly, the same problem. Even tried adding hostnames to the filter vice using wildcard. Same. The odd thing. The DS reports that Windows hosts are running the Linux TA, but when you check the Windows hosts, they are running the Windows TA as they should be

Re: Deployment Server clients have wrong apps

isoutamo — Wed, 27 Mar 2024 19:34:20 GMT

Have you local indexes on DS or are you sending logs to your real indexers? This has changes on 9.2.x and it could cause something weird.

Re: Deployment Server clients have wrong apps

tlmayes — Wed, 27 Mar 2024 19:35:54 GMT

I opened a P2 3 days ago... still waiting. Typical

Re: Deployment Server clients have wrong apps

tlmayes — Wed, 27 Mar 2024 20:04:26 GMT

Great point, and something I did not know beforehand. In troubleshooting stumbled onto the documentation stating what you are pointing out, the new _ds* indexes. So yes, the _ds* indexes are local to the DS.

Re: Deployment Server clients have wrong apps

isoutamo — Wed, 27 Mar 2024 23:16:26 GMT

Based on docs this should works. BUT on example part those platform selections have done on app not serverclass level. Maybe you should try that?

Btw have you configured this by gui or manually with text editor?

Re: Deployment Server clients have wrong apps

tlmayes — Thu, 28 Mar 2024 11:30:49 GMT

Such a small and straightforward environment I used GUI. I get the sense there is a bug in 9.2.0.x

Re: Deployment Server clients have wrong apps

isoutamo — Thu, 28 Mar 2024 15:29:21 GMT

Ok, then it’s best to wait resolution for your P2 case.

Re: Deployment Server clients have wrong apps

tlmayes — Wed, 03 Apr 2024 15:31:09 GMT

Splunk support concluded it was an "as yet discovered software bug"

Re: Deployment Server clients have wrong apps

hmallett — Thu, 25 Apr 2024 09:45:21 GMT

I have nothing to add, except to say that I have observed the same bug, where the server classes that use machine filtering display the incorrect clients in the UI.

The bug remains in version v.9.2.1

Re: Deployment Server clients have wrong apps

hmallett — Thu, 25 Apr 2024 13:28:01 GMT

I believe that this bug is planned to be fixed in 9.2.2

Re: Deployment Server clients have wrong apps

mortification77 — Tue, 04 Jun 2024 15:15:27 GMT

From what *I* have seen, the machineTypesFilter seems to be at the root of this bug.

This is the absolute *WORST* update that I've seen in the last 6 years that I've been working with Splunk.

I did read something that would indicate that the (white|black)list.X can also take OS Strings, but the docs call it "platform dependent", so I am putting it off until we can actually SEE what's being deployed again.

I have noticed yet *ANOTHER* bug... After a Deployment Server has been running for a bit, ANY CALL that would query DS Client information will TIMEOUT.

I have multiple scripts that read data from /services/deployment/server/clients, and I've bumped the timeouts to 30 seconds, and it still times-out. It used to take < 2s to pull data from THOUSANDS of clients.

Re: Deployment Server clients have wrong apps

hieuba6868 — Thu, 03 Oct 2024 12:25:13 GMT

Hi @tlmayes,

i have a same issue.

Any solutions for this? Pls

Re: Deployment Server clients have wrong apps

tlmayes — Thu, 03 Oct 2024 13:13:00 GMT

Has been officially registered as a bug. No ETA on fix