Best practices are to create your app on a standalone search head, and once you have it created and configured, use the deployer to push it out to the Cluster.
I can't just create it on the deployer? I'm not talking about a fully functional app - basically just a template for customers to begin adding reports/dashboards.
You can create it on the deployer. But consider for it to be pushed out to the SHC members, it has to be moved from the $splunkhome/etc/apps to the $splunkhom/etc/shcluster folder on the deployer, and then deployed.
In search head clustering, you must use the deployer to push any non-default apps to the search heads. You cannot install apps directly on the individual search heads.
Where you initially create and test the app, pre-deployment, is your call. As esix_splunk suggests, the best approach is to do so on a standalone search head.
For more information on pushing apps via the deployer, see http://docs.splunk.com/Documentation/Splunk/6.2.2/DistSearch/PropagateSHCconfigurationchanges