I'm currently working on setting up splunk which I have done but was asked for a setup that I have not done or attempted before and was curious about any thoughts somone could provide. They are asking me to setup a splunk cluster made up of multiple splunk searchhead/indexer instances. Essentially, we have the master splunk that over sees the whole system, and slave splunks that over see the subsystem. Those individual subsystems splunk data, would need to replicate to the master system splunk but not replicate back. For example:
Splunk slave 1 collects logs from its machines, and replicates to master splunk
splunk slave 2 collects logs from its machines and gets replicated to master splunk
Master splunk gets all this data but none of it gets replicated back so that the slave splunks do not contain one anothers data. The master would be a infrastructure wide instance able to view data across all systems while the slaves can only view its local systems data. Thats why each would have to have their own search head.
If i point to different indexers, I read it will count twice against the licensing. Replication gets around this but I have not found if you can setup one way replication so that only master splunk gets all the data while the local splunk can only see its own. Everything seems that if i enable replication, slaves would send to master, and master would replicate any difference in data to each one and that defeats the problem of keeping the slaves data separate.