Run indexes on different servers

[carljohan]

We are planning to have a Splunk setup where we have:

• 1 server running a Splunk indexer
• 2 servers per operation from which log files are forwarded by universal indexers

We are then planing to store each operations index on the operations own server instead of on the Splunk indexer.
All servers are on the same network.

Is this setup doable?

[woodcock]

What you are actually desiring is known as Mutitenancy. Check out these links (or do your own search):

http://docs.splunk.com/Documentation/Splunk/latest/Updating/Deployinmulti-tenantenvironments

https://wiki.splunk.com/Community:Multi-tenant_scenario

https://answers.splunk.com/answers/127073/indexing-best-practices-for-multi-tenant-environment-and-v...

http://blogs.splunk.com/2011/11/04/splunk-and-multitenancy/

[woodcock]

The only way to "do" it this way is to somehow have your Forwarders' storage directly accessible by your Indexers which I do not see how you will be able to do it other than if you use NFS and mount to both systems. This means your Forwarders will host the Indexer's write operations only. The other way to make it work is to configure your Forwarders to also be Indexers which I think is a VERY bad idea because the Indexers are the ones that always (eventually) get overloaded with work and slow to a crawl which will adversely impact the job that those servers are (primarily) supposed to be doing. Why are you using such a strange (bad) constraint? Anyway you do this, it is a pretty bad idea and, to quote a splunk T-shirt, you are "looking for trouble".

[carljohan]

Thank you for your feedback.

The data from the different operations need to be separated due to contractual reasons.
Any suggestion on how we can accomplish that in a less strange (bad) way?

[woodcock]

See my latest answer regarding Multitenancy.

[muebel]

Hi Carljohan, You would not want to necessarily forward the DB logs to the central Splunk Instance. You could have op1 and op2 DBs just index locally, and then set the central Splunk instance up to search each of the DB instances, while still forwarding the app boxes to the central Splunk server. Other than that I can't detect any issue with this setup.

Let me know if this helps 😄

[carljohan]

Thanks Muebei. But that would require us to do a full Splunk install on the DB servers, not only a universal forwarder. Right?

[muebel]

Yup, the universal forwarder doesn't have any local indexing capability, it can only forward events.

[aholzer]

The indexer needs direct access to the location where you are going to store the data. So if you are in a Linux environment, you'd have to mount the drive from the "DB Servers" on your Splunk indexer server(s). Then make sure you define in indexes.conf the correct paths using the mounts.

Hope this helps