I am wondering whether they are any security issues/concerns associated with deploying an instance a Splunk Enterprise server in a DEV environment that has access to logs stored in the PROD environment and vice versa.
Should deployment instances and the associated log sources be deployed separately in DEV and PROD environments with limited interconnectivity (there is no firewall filtering traffic from DEV to PROD).
It's very specific to the organization, but most companies consider accessing PROD data from DEV servers as a security violation. This is applicable to any product/tool, not specific to Splunk.
I agree with what has been said previously about DEV and PROD environments being segregated, and typically by security policy, not allowed to talk to each other.
However, I have also been in environments where it isnt possible to have the same data sources or sourcetypes available in both environments. So in these use cases, a hybrid/distributed search configuration is one approach to this.
That being said, there are some concerns, mainly in regards to potential service impact of the DEV environment against PROD. With the way distributed search works, its possible that the DEV environment could steal resources from the PROD environment.
I think the best approach would be to replicate indexed data from PROD to DEV. If you are unable to replicate the data sources, you can always copy the indexed warm/cold buckets over to DEV, and this would give a valid data set to work on. It wouldnt be real time, but it would still work.
Another option would be Eventgen, and generate your own events based on your own sourcetype requirements. Customize it so it would resemble your own environment...