Archive

Indexing best practices for multi-tenant environment and various APPS

bkcarter
Path Finder

I would like to have a discussion about the best practices relating to a multi-tenant environment and how to best create access control as well as use off the shelf apps.

Background: I work for a company that has 7 subsidiary companies. I am in charge of security monitoring for the entire corporation. We have deployed Splunk indexing and searching at the top corporate level (Linux based). Each of the 7 subsidiaries feed their log data to this centralized Splunk system via intermediate forwarders. Bandwidth is not an issue and this works well.

Each of the 7 subsidiaries has their own types of networks and equipment. Some are Windows based, others Linux and Mac. Some use one brand of firewalls while others use different brands. Some of this equipment has apps built by their respective vendors that can be installed on Splunk.

In order to provide access control for the 7 companies, I have created an index for each company's data. Roles are then created for the users in those companies and they are restricted to only see the data in their respective index.

This has worked great so far, and I have began to build apps and dashboards for them.

HOWEVER…. When I go to install an APP such as Palo Alto's app, Cisco's app, or Splunk for *nix apps I continually run into the fact that they are hard coded to use specific indexes. Some of them allow you to change the index in the settings, but I want to be able to use the app against the company indexes that I have defined.

I do use the deployment server and have a set of apps that can be deployed depending on the criteria there. I know that I can get right down into the code of the apps and tweak a lot of things, but I worry about the ability to keep upgrades simple in the future.

My recent experiences with apps has me questioning whether the separate indexes is a good idea or not. The major benefits of them is that each one can have it's own retention (hot to warm, etc) settings, and that access control is easier.

It does not seem that current day apps are designed with this kind of structure in mind.

Please share with me your experience and help me understand the pitfalls of the direction I am currently going, or how I can make it more scalable and usable for the various levels. How do you do it?

Thanks!

Tags (3)
1 Solution

Ayn
Legend

Automatic upvote for question regarding multi-tenant deployments... 🙂

We're in a pretty similar situation as yours in that we have totally different "customers" with completely separate environments, probably even more so than in your case, so even IP addresses, hostnames, critical assets, privileged users etc could theoretically have the same name at two different customers but still be completely different things. Yet we run most things from central search heads.

Unfortunately I don't have a "silver bullet" for you in how to deal with this - I can only verify that multi-tenancy with Splunk IS a pain and the more you look at it the more it seems like doing multi-tenancy is really bending how you're supposed to use Splunk.

In the case of apps that explicitly use a certain index - yes, you will have to rewrite the app to take that away, and instead set the app to not care about the index at all, using each user or role's index access to define which data will be visible to which user. For users with access to several customers' data (like your own admin user) you'd have to rewrite your app to include something like dropdowns for choosing the correct index.

With more and more apps making use of Splunk's high performance datastore via tsidx files this becomes tricky as well, as these have separate permission models - I'm actually unsure if Splunk is enable to enforce access to these at all right now.

It gets hairier if you want to run environment-dependent lookups too, but I don't know if you're trying to do this right now. In short, this becomes a problem if you want to use the same lookup for different environments and have different environments return different data.

Sorry for not giving you better news...

View solution in original post

satishsdange
Builder

Apps are not designed to support multi-tenancy. Have a look at this presentation. This will address your most of queries.
http://conf.splunk.com/sessions/2014
Multiple, Auto-deployed, Splunk Instances on a Mission-critical Managed Service Provider Cloud

0 Karma

mikaelbje
Motivator

Back in 5.X this was possible using tenants.conf where you could point a subset of hosts to another deployment server based on their IP/DNS with wildcards: http://answers.splunk.com/answers/100554/can-a-deployment-server-direct-client-requests-to-another-d...

You could also use repositoryLocation per app per serverclass in 5.0 which was very useful.

It had the following benefits:
* Same deployment app names in separate subdirectories could be used across the divisions/subsidiaries and you could do local changes in the local/ dirs. Allows for easier app upgrades.

Drawbacks:

  • tenants.conf: No single pane of glass to manage ALL deployment clients. You'd need to log into each deployment server to manage their server classes, clients and apps
  • Per serverclass per app repositoryLocation: No GUI based serverclass configuration

However, as of 6.X tenants.conf and the per serverclass per app setting have been removed which means you now need to think differently. Possible solutions would be:

  • iptables to forward data to another deployment server depending on source IP: Not a good idea as it removes visibility from a Splunk admin's perspective
  • Copying and renaming apps: Would require you to keep track of "master" and "slave" versions in a document.

I'm uncertain of why tenants.conf was removed and I'm hoping that this will get reintroduced, perhaps in an even smarter way. Virtual containers in the deployment server would be AWESOME as you'd have a single pane of glass and could use the original app names multiple times, just in separate subdirectories.

Maybe the next version of Splunk contains improvements in this area. I'm thinking of distributed deployment servers, virtual containers etc for scalability. Perhaps the official Splunk best practice has changed to use Puppet or another configuration management tool for larger environments. That would again remove visibility for the Splunk admin.

Any official Splunk comments? Maybe @yannKk knows? 🙂

0 Karma

Ayn
Legend

Automatic upvote for question regarding multi-tenant deployments... 🙂

We're in a pretty similar situation as yours in that we have totally different "customers" with completely separate environments, probably even more so than in your case, so even IP addresses, hostnames, critical assets, privileged users etc could theoretically have the same name at two different customers but still be completely different things. Yet we run most things from central search heads.

Unfortunately I don't have a "silver bullet" for you in how to deal with this - I can only verify that multi-tenancy with Splunk IS a pain and the more you look at it the more it seems like doing multi-tenancy is really bending how you're supposed to use Splunk.

In the case of apps that explicitly use a certain index - yes, you will have to rewrite the app to take that away, and instead set the app to not care about the index at all, using each user or role's index access to define which data will be visible to which user. For users with access to several customers' data (like your own admin user) you'd have to rewrite your app to include something like dropdowns for choosing the correct index.

With more and more apps making use of Splunk's high performance datastore via tsidx files this becomes tricky as well, as these have separate permission models - I'm actually unsure if Splunk is enable to enforce access to these at all right now.

It gets hairier if you want to run environment-dependent lookups too, but I don't know if you're trying to do this right now. In short, this becomes a problem if you want to use the same lookup for different environments and have different environments return different data.

Sorry for not giving you better news...

View solution in original post

bkcarter
Path Finder

Drat! I really was hoping that I was just missing something!

I guess that will make the learning curve even steeper if I want to get the benefits of the various Apps. I will just use some of the apps as a framework to build my own.

It would be nice if you could define a "virtual" index type of mechanism that would allow you to make an app "think" it was using it's own index, but really pointing to something else that is "defined" somewhere.

Thanks for the response. It is what I expected, but had to ask before I stepped into the next phase.

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!