What is the best practice to manage the same index with the same log paths but in different environments?

[mudragada]

I have a clustered Splunk env with an index="myjavaapp".
I need to collect the logs from multiple environments - Dev/QA/Stress/Pre-Prod/Prod - where each environment has about 2 to 15 servers. The logs on all servers fall in the same path /opt/appserver/logs/server.log.

What is the best practice in this case to maintain logs from different envs, but coming from the same path.

Also, I have other applications coming from the same environments, but different servers. I need to implement the strategy for all the indexes. (ex: index="mywebapp")

[DalJeanis · Accepted solution]

One simple way would be to set up your configs to (at index time) override the index or sourcetype for those logs based on the server/host, and append dev_ or qa_ onto the index name. If your host boxes are consistent in their usage (prod, qa, dev), you could configure that consistent prefix for EVERY index, which would go a long way toward making the system bulletproof.

Given the general InfoSec maxim called "the Principle of Least Privilege", employees should generally not be able to see information that they have no business need to see. That means that employees who work solely in the dev and qa areas should not have access to the prod data, and vice versa. Indexes and role-based security is the standard splunk way of achieving that data-level InfoSec.

[adonio]

Hi mudragada,
can you elaborate on your question, are different logs (sourcetypes) are written to same log (source) and being sent to the same index? what would you like to achieve? seperate environment data in separate indexes? renaming sourcetypes?
this answer is a good place to start: https://answers.splunk.com/answers/41040/filtering-data-into-different-indexes.html

[mudragada]

I referred to the answer you pointed to. Let us consider an example:

I have a java application running in servers 1&2(dev), servers 3&4(qa), servers 5&6(Pre-Prod), servers 7,8,9,10,11(Prod). The log path is the same across all the servers. /opt/appserver/logs/server.log.

Now, I have an index="myjavaapp" which is getting logs from 7,8,9,10,11 servers of PROD. But for dev purposes and other test purposes, I need to index the rest of the logs as well into Splunk Enterprise. I have about 300 splunk search users - who are about to use these logs.

I'm able to do it in index="myjavaapp" itself - which has a collective size now for the index. Also, there was a proposal to make indices like "dev_myjavaapp", "qa_myjavaapp" - but is a little tedious when it comes to all envs.

That is where i'm looking for a solve - where the practice can be established throughout and can be communicated for the 300+ users on how they should carryout their search across all envs.

[adonio]

you can filter your searches by host (server) for example: index = "myjavaapp" host = server1 oR host = server2. Having the same source (if i understand correctly its just the same path on each sever) does not limit you as you can filter by host. Now, its is up to you if you want to have lots of data in one index or divide to many indexes, there considerations for both approaches. For the users, you can use tags, eventtypes and macros for example to help them carry searches per environment.
Hope it helps