topic Run indexes on different servers in Getting Data In

Run indexes on different servers

carljohan — Mon, 19 Oct 2015 12:27:29 GMT

We are planning to have a Splunk setup where we have:

1 server running a Splunk indexer
2 servers per operation from which log files are forwarded by universal indexers

We are then planing to store each operations index on the operations own server instead of on the Splunk indexer.
All servers are on the same network.

Is this setup doable?

Re: Run indexes on different servers

aholzer — Mon, 19 Oct 2015 12:50:29 GMT

The indexer needs direct access to the location where you are going to store the data. So if you are in a Linux environment, you'd have to mount the drive from the "DB Servers" on your Splunk indexer server(s). Then make sure you define in indexes.conf the correct paths using the mounts.

Hope this helps

Re: Run indexes on different servers

muebel — Mon, 19 Oct 2015 13:07:59 GMT

Hi Carljohan, You would not want to necessarily forward the DB logs to the central Splunk Instance. You could have op1 and op2 DBs just index locally, and then set the central Splunk instance up to search each of the DB instances, while still forwarding the app boxes to the central Splunk server. Other than that I can't detect any issue with this setup.

Let me know if this helps 😄

Re: Run indexes on different servers

carljohan — Mon, 19 Oct 2015 13:17:40 GMT

Thanks Muebei. But that would require us to do a full Splunk install on the DB servers, not only a universal forwarder. Right?

Re: Run indexes on different servers

woodcock — Mon, 19 Oct 2015 13:20:43 GMT

The only way to "do" it this way is to somehow have your Forwarders' storage directly accessible by your Indexers which I do not see how you will be able to do it other than if you use NFS and mount to both systems. This means your Forwarders will host the Indexer's write operations only. The other way to make it work is to configure your Forwarders to also be Indexers which I think is a VERY bad idea because the Indexers are the ones that always (eventually) get overloaded with work and slow to a crawl which will adversely impact the job that those servers are (primarily) supposed to be doing. Why are you using such a strange (bad) constraint? Anyway you do this, it is a pretty bad idea and, to quote a splunk T-shirt, you are "looking for trouble".

Re: Run indexes on different servers

carljohan — Mon, 19 Oct 2015 13:28:45 GMT

Thank you for your feedback.

The data from the different operations need to be separated due to contractual reasons.
Any suggestion on how we can accomplish that in a less strange (bad) way?

Re: Run indexes on different servers

muebel — Mon, 19 Oct 2015 14:02:38 GMT

Yup, the universal forwarder doesn't have any local indexing capability, it can only forward events.

Re: Run indexes on different servers

woodcock — Mon, 19 Oct 2015 15:02:44 GMT

What you are actually desiring is known as Mutitenancy. Check out these links (or do your own search):

http://docs.splunk.com/Documentation/Splunk/latest/Updating/Deployinmulti-tenantenvironments

https://wiki.splunk.com/Community:Multi-tenant_scenario

https://answers.splunk.com/answers/127073/indexing-best-practices-for-multi-tenant-environment-and-various-apps.html

http://blogs.splunk.com/2011/11/04/splunk-and-multitenancy/

Re: Run indexes on different servers

woodcock — Mon, 19 Oct 2015 15:03:14 GMT

See my latest answer regarding Multitenancy.