I have Splunk distributed 7.2.1 (1 dedicated Search Head with multilple non clustered indexers)
Note: the dedicated search head is acting as deployment server and license manager as well.
This splunk doc might give you an idea on scaling your infrastructure..
http://docs.splunk.com/Documentation/Splunk/7.2.1/Capacity/Summaryofperformancerecommendations
Out if curiousity, would the remote indexer you're planning on better suited as a heavy forwarder that just sends traffic back to your main indexers at the other end of the VPN tunnel? Seems like the easiest way to avoid latency issues depending on the distance.
When the client chooses that the data stays stored locally at his side we can't judge him! also i would need another indexer at my side to receive parsed event coming from the heavy forwarder as you suggested!
My point here that i would search data through my dedicated Search Head (no local storage/index) to the indexer(at the client side) via VPN, data will be stored there, his license will be a slave of my license manager
The Search head uses "URL:8089"
As long as it can connect to it should not be any problems.
But the underlaying infastructure may be using som CPU resources. Is the VPN a software or is it Site2Site vpn.
You will most likely encounter some lag, but I think it wil work.
But it all depends on network and the cpu/memory of the Search head.
The indexer you are pulling data from wil not even know if youre on another network.
But if you can reach the indexer/peer via the VPN tunnell you are all good.
The Search Head is deployed with the best practice hardware requirements so i don't think there will be a problem there.
The best way i'm planning this is on a Site2Site VPN, i think the network capacity is extensible so logically there is no problem.
Please let us know how it went, if you are experiencing lag then you could adjust the timeout on the Search head.
It's working for 2 months now !
no break ups or failures
Wonderful:)
This splunk doc might give you an idea on scaling your infrastructure..
http://docs.splunk.com/Documentation/Splunk/7.2.1/Capacity/Summaryofperformancerecommendations