For indexers, if you're using a master server, you really should use the master to control your configs. How you control your master is up to you - if you want to use helm, jenkins, whatever, that's fine - just use the Splunk commands to push those configs to the indexers. This ensures that the configs are consistent and you can also control reboots somewhat more easily.
Search heads are similar - if you're using a search cluster, push your configs to the deployers using your tool of choice, and use the Splunk cli commands to push the configs out to the search cluster. Individual search heads can be configured any way you like.
I have mixed feelings about using configuration tools to manage forwarders. I've found it's a lot harder to ensure configurations if you have any questions about your config management. I've seen chef deployments where there were 100's of non-compliant forwarders because of chef issues, and if they were all pointing to a deployment server, you would then have a centralized place to look at what Splunk is up to. Now is the deployment server perfect? Far from it; it has a lot of shortcomings. But in general it's the easiest choice to start out with. You might want a base config that you push to the forwarders for initial installation and then push the rest from a deployment server. Be careful in that type of scenario - you can accidentally have competing configuration management systems.
Remember Splunk is mostly stateful (the master is completely stateless, but the other systems keep varying degrees of state) so things like containers, autoscaling groups in AWS, etc, where you swap in/out base systems is not really ideal with the current Splunk design. Maybe in the future, if they consider containerization in their design, then it will be easier to containerize.
