That is an interesting issue since  you will find everything ok  related to configuration and best guess is that you are hitting the maximum limit of knowledge bundle replication and max_content_length   Below is the new recommended setting as per splunk document  on  search head distesearch.conf [replicationSettings] maxBundleSize = 4096   you must also increase max_content_length in server.conf on the indexers (search peers) to at least 4GB and also on search head and deployer.  [httpServer] max_content_length = 4294967296 ... View more

If you are using deploment server to create the token and push it to your heavy forwarders where it should be actually authenticate then you have to: 1. To make sure you change useDeploymentServer flag to true as below.   useDeploymentServer = 1   When this option is set to 1 and you make UI-based HEC changes on the deployment server, those changes are placed directly in the $SPLUNK_HOME/etc/deployment-apps/splunk_httpinput/ folder, rather than in $SPLUNK_HOME/etc/apps/folder.  Because if  your inputs changes is there in the $SPLUNK_HOME/etc/apps/<anyapp>/inputs.conf  on deployment server and also in your Heavy forwarder . Then the rest/curl call to token will end up in Invalid token response code 4. ... View more

If you are using deploment server to create the token and push it to your heavy forwarders where it should be actually authenticate then you have to: 1. To make sure you change useDeploymentServer flag to true as below.   useDeploymentServer = 1   When this option is set to 1 and you make UI-based HEC changes on the deployment server, those changes are placed directly in the $SPLUNK_HOME/etc/deployment-apps/splunk_httpinput/ folder, rather than in $SPLUNK_HOME/etc/apps/folder.  Because if  your inputs changes is there in the $SPLUNK_HOME/etc/apps/<anyapp>/inputs.conf  on deployment server and also in your Heavy forwarder . Then the rest/curl call to token will end up in Invalid token response code 4.   ... View more