One index per source?

[leune]

After having used Splunk for a few months now, more and more people are requesting access to it. Great! That's exactly what I hoped would happen! However, with doing that also comes the point where we will have to start thinking about segmenting my data into multiple indexes for access control principles.

In my environment, Splunk is predominately used for monitoring syslogs, application logs and windows event logs.

How bad of an idea is it to create one (or maybe even more) indexes per source host? Alternatively, I can create one index per functional group to limit access, but since ownership of logs may change, I'd rather figure out the access control stuff via roles than by indexes.

Are there any expected (or unexpected) side-effects of creating a relatively large number of indexes (probably over 100)? I'll use a naming scheme that will allow me to search multiple indexes at once (ie: host_, app_, or whatever may make sense). Does multiple indexes have an effect on search performance? I have 8 cores at my disposal.

[kristian_kolb · Accepted solution]

As sdaniels says, it would probably be better to group stuff together based on something like;

• windows servers application logs (app mgmt team)
• os logs (server team)
• core network stuff from routers, switches etc (network team)
• firewall traffic logs (security team)
• windows clients eventlogs (helpdesk) etc

i.e. what would constitute reasonable groups which correspond to people working in the organization.

Another thing to take into consideration is the retention time, which is set on a per-index level. So if some app logs need to be stored for 5 years, and other for only 3 months, you'd need to split them into two indexes. Otherwise you'll end up storing the short-term logs for 5 years as well.

So basically, create a matrix of retention requirements and access control requirements, which should give you the number of indexes required. If you end up with more than 20, you should probably seek permission to alter the requirements, since micromanaging disk utilization and indexes filling up is harder with many indexes.

Hope this helps,

Kristian

[leune]

That's the direction I was going in also; data retention and data block signing requirements are probably going to be the decisive factors in index selection.

Access control is something that I can probably fix using some well-chosen tags and by using custom roles with restricted search terms.

Thanks for the feedback!

[sdaniels]

I don't believe it would be recommended to create an index for every source. If you create an index for the functional groups you can still change and limit user access based on pre-prending a search to a role like host=dev* or host=dev1 host=qa2 etc.