Why do the multiple deployment servers behind Load Balancers, have different checksum values?

[sylim_splunk]

We have 8 deployment servers. Since some time back we started to have symptoms like checksum mismatch in splunkd.log on the universal forwarders, which causes UF to keep downloading apps. It is certain that no changes has been made to those apps.
We are using crossServerChecksum = true as recommended in the answers link;

https://answers.splunk.com/answers/113792/multiple-deployment-servers-checksum-mismatch-among-instan...

But still, all deployment servers appear to have different checksums by looking at the logs in universal forwarders.

[sylim_splunk · Accepted solution]

As crossServerChecksum = true is configured, splunk will not consider timestamp or permissions of files in calculating checksums, there is no manual way of simulating the checksum calculation exactly as splunk does as of this writing but you can briefly check using md5 as below;

splunk cmd openssl md5 "tarball of the app"

Also make sure there is no hidden files or directories which can cause different checksums. Below has .git which caused different checksums.

-rw-r--r-- 1 support support 0 Jan 31 14:07 README.md
drwxr-xr-x 7 support support 4096 Feb 26 13:59 .git
drwxr-xr-x 2 support support 4096 Feb 26 13:59 auth
drwxr-xr-x 2 support support 4096 Feb 26 13:59 local