I agree with what has been said previously about DEV and PROD environments being segregated, and typically by security policy, not allowed to talk to each other. However, I have also been in environments where it isnt possible to have the same data sources or sourcetypes available in both environments. So in these use cases, a hybrid/distributed search configuration is one approach to this. That being said, there are some concerns, mainly in regards to potential service impact of the DEV environment against PROD. With the way distributed search works, its possible that the DEV environment could steal resources from the PROD environment. I think the best approach would be to replicate indexed data from PROD to DEV. If you are unable to replicate the data sources, you can always copy the indexed warm/cold buckets over to DEV, and this would give a valid data set to work on. It wouldnt be real time, but it would still work. Another option would be Eventgen, and generate your own events based on your own sourcetype requirements. Customize it so it would resemble your own environment... ... View more

Here's a good guide on how to install Splunk securely.. This will show you how to configure role-based access control, authentication LDAP, Single Sign On with SAML, configuring SSL, and installing Splunk Enterprise securely. http://docs.splunk.com/Documentation/Splunk/6.4.2/Security/InstallSplunksecurely ... View more