Deployment Server throwing error when making changes to an app and reloading the server class

[clozach]

Reference to https://answers.splunk.com/answers/666034/forwarder-management-warning-icon.html because it was never really answered.

Running 7.2.1, Distributed Deployment, 4 Indexers Clustered, License server/cluster master, deployment server/monitoring console, Splunk SH, Splunk ES SH 5.2.2

I am receiving a red ! next to the app in the serverclass after making changes to an app. I also tried adding a new app and it received an error. The _internal logs just keep showing failed installation with the checksum but do not give me a reason for the failure. I am also receiving an error when searching displaying the following.

Could not load lookup=LOOKUP-CategoryString_for_windows
Could not load lookup=LOOKUP-app4_for_windows_security
[indexer1] Could not load lookup=LOOKUP-CategoryString_for_windows
[indexer1] Could not load lookup=LOOKUP-app4_for_windows_security
[indexer2] Could not load lookup=LOOKUP-CategoryString_for_windows
[indexer2] Could not load lookup=LOOKUP-app4_for_windows_security
[indexer3] Could not load lookup=LOOKUP-CategoryString_for_windows
[indexer3] Could not load lookup=LOOKUP-app4_for_windows_security
[indexer4] Could not load lookup=LOOKUP-CategoryString_for_windows
[indexer4] Could not load lookup=LOOKUP-app4_for_windows_security

This might be due to a separate issue of incompatible apps, but I can't push anything to my indexers when the deployment server can't push apps.

Thanks for your time.

[clozach]

Correction it seems to just be the cluster master apps serverclass that is failing. This server class just consists of the cluster master and then should deal the app down to the indexers.

[skalliger]

There are no serverclasses when we are talking about the Cluster Master. Serverclasses belong to the Deployment Server. Are your lookups in the app actually being distributed to the indexers? Maybe they're blacklisted: https://docs.splunk.com/Documentation/Splunk/7.2.5/DistSearch/Whatsearchheadssend

It sounds like you are hosting the CM and the Deployment Server on the same machine. You should not do that.*

In bigger environments definitely not:
https://docs.splunk.com/Documentation/Splunk/latest/Deploy/Manageyourdeployment#Whether_to_colocate_...

Skalli

[clozach]

Hi Skallinger,

No the CM and the Deployment Server are not the same machine. What I was saying is the serverclass having the issue was called cluster_master_apps and has all indexer apps with the cluster master as the client attached to that server class. So somewhere in our PS he set this up and now somewhere between the deployment server and the indexers, the app is not being pushed out and saying install failed. I believe in our PS engagement he set it up so the CM would push the apps through to the indexers on the slave-apps directory. I could be wrong, but if you have any more info that would be helpful I'd appreciate it. At least from a debugging standpoint. I've had a case open with Splunk for almost 2 weeks they seem to also be stumped.

Thanks,
Christian