I have something working and I'm not sure why it's working.
All of my forwarders are now set up as deployment clients. And all seems to be working correctly. Except I just found an error in my server class "production_servers" so it was not getting deployed. And that's where I set an index=prod. I can't find any other server classes where the index is set to prod. Yet all the appropriate events are still being put in the prod index.
Is this configuration coming from the old $SPLUNK_HOME/etc/system/local/inputs.conf file on the production forwarders? I did not remove those files from any of my deployment clients. I saw nothing explicit in the documentation to tell me to do something.
Is that file taking precedence over what the deployment server is sending? All guidance appreciated.
If you're deploying your configuration via a deployment server, I believe the config files you are editing on a deployment server should be pushed to etc/apps/deploymentapp, not under etc/system/local. I know in my environment, I don't have anything more than the "[default] host=hostname" in etc/system/local/inputs.conf file, and specify the other items (index=indexname) n the etc/apps/deploymentapp/local/inputs.conf. I would assume(and maybe i shouldn't) that your deployment app should be deployed to etc/apps/production_servers/local/inputs.conf. If your etc/system/local/inputs.conf specifies "index=prod", then I believe it will default all data to that unless otherwize specified in the deployment app in etc/apps/production_server/local/inputs.conf
To summarize, you DO think I will have to remove etc/apps/deploymentapp/local/inputs.conf on the clients? I guess that makes sense. I wish it had been explicitly called out in the documentation.
This probably goes to configuration file precedence which I am certainly not an expert on.