Indexes and Server types

[texascj]

I have about 100 servers.  These are a mix of different Oracle servers, Databases, Web Apps Servers, Data Warehouse servers, SSO Servers and OBIEE servers.  Of these, there is also the standard Dev/Test/Prod environments and this is all supporting 5 different development / sustainment projects.

A request was made to our Splunk Admin in the form of the Server name and all of the log files our engineer could think of at the time.  It appears the Splunk Admin just crammed everything into a single index.  Literally hundreds of log files as each server appeared to have 10-15 log files identified.

Given the servers do different things, the request didn't necessarily have the same log files identified for every server.  I would have "expected" the request would have been vetted to answer "What do you really need?" rather than "HERE YOU GO!"  Maybe I've done Software Development too long, it could be me.

Anyway, was this the right way to go?  Would it have made more sense to have 1 index for the Database Servers, 1 index for the Web App Servers, 1 index for the Data Warehouse, etc...?  Or, perhaps 1 index for the Production assets and 1 for Test and 1 for Dev?

There doesn't appear to be a "best practice" that I can find...and what I have is ONE FREAKING HUGE index.

If you read this far, thanks.  If you have a cogent answer that makes sense to me, even better!

[texascj]

Okay, if I understand the two of you correctly then I'm in a pickle...not a fan of swimming in the brine so to speak.  There was no "architecting" or "engineering" or "questioning" involved so I suspect the Splunk Admin presumed we knew what we wanted.  I walked into this cold and after the fact w/ a "make it work" directive.

Keep in mind, two weeks ago I could spell "Splunk" but had never used it.  I find it quite complicated.

Soooooooo, if I were to start over what would be the needed requirement one must address w/ respect to having the index(es) built?  From the above I can see we should have addressed :

1.  Data Retention period by Dev/Test/Prod 

2.  Access restrictions (if any) by the same

3.  What are you wanting to measure?  (implies "knowing" the data)

4.  Age / staleness of data allowed

5.  Volume size of the ingested logs per server that are used to build the index(es)

Would there happen to be a "best practice" approach that should be used, or should have been used, regarding the initial build of the environment?

Thanks all!

[PickleRick]

Well... the typical approach is to get a piece of paper (or some excel sheet) and list all (kinds of) sources you're gonna be getting the events from. And work with that - think who should have access to that data, how much of that data you're gonna need, for how long it will have to be stored, what are use cases you'll be implementing using this data (because that also can affect the data distribution).

It's a bit complicated (and that's why Splunk Certified Architect certification is something you get only after you've already certified for Admin and while the course itself might be taken before that there's not much point in doing so) and it's hard to say in a short post about all possible caveats regarding proper indexes architecture.

But bear in mind that indexes in Splunk do not define the data contained within them in any way (at least from the technical point of view). They are just... "sacks" for data thrown in. And you can - if needed - have several different "kinds" of data within one index and usually (unless you have some strange border case) it doesn't matter and doesn't give you a significant performance penalty. You simply choose some of that data when searching by specifying metadata fields (like host, source, sourcetype) as your search terms. One exception here (which I'll not dig into since we're apparently not talking about it at the moment - there are two types of indexes - event indexes and metrics indexes).

[ITWhisperer]

If you were starting over, or even just looking to refactor what you have, number 3 is the most important question. What do you (and your users) want to get out of putting all this data into Splunk. The other questions are supplementary to this. Establish some short term goals and start building / refactoring small and build capabilities and features that your users want, let them guide you.

[PickleRick]

While the admin might have asked if that's really what you want, that's more of an architect's job to design your indexes properly.

As for splitting the data - there are usually two main reasons for splitting data into separate indexes - retention parameters and access restrictions. Just because the servers are db servers or just because they are dev servers doesn't mean that you need separate indexes. You might though if separate teams need access to logs from dev/testing/prod environments or if you need to keep data from dev for a month but from prod for two years. A good architect will also try to find out if there is a chance of a need for such differentiation in forseeable future.

Another thing that could warrant separate indexes is huge difference in volume of data between sources.

But all those things need to be considered on a per-case basiss. There is no one-size-fits-all solution saying how you should split your data between indexes.

[texascj]

Well....I suppose "best-practice" would have been a better tag.  Go figure...

[ITWhisperer]

One of the key attributes of an index is the retention period, so, assuming you would like to retain different sorts of information for different periods of time, then you should consider putting them into different indexes. For example, you might want to consider keeping production information for longer than development. The different types of logs can go in the same index but the key here is to use different sourcetypes so they can be distinguished and treated differently, e.g. field extractions. So, you are right, your Admin should have asked questions like what do you want to do with the data, how long to do you want to keep it, etc. Having said that, and since you have added the summary indexing tag, you could run reports on the large index to split the useful data off into summary indexes, but then it depends on how timely you need the data e.g. as soon as it hits the index or only after the summary index report has been executed.