topic Re: Separating logs from different environments in Knowledge Management

Separating logs from different environments

nembela — Tue, 07 Dec 2021 09:47:56 GMT

Hi,

Till now we only collected logs from production servers with Splunk. But soon we will onboard the system logs from non-prod (Linux, Windows) servers.

What is the best way to differentiate between the logs from different environents?

different index? All these logs have the same retention time
different sourcetype? All the logs are system logs (Windows, Linux)
eventtype?
a dedicated "environment" field?
tagging?

Thanks,

Laci

Re: Separating logs from different environments

gcusello — Tue, 07 Dec 2021 09:55:41 GMT

Hi @nembela,

at first the choose of an index depends on two reasons.

retention,
accesses.

usually non prod logs have a different retention and different access grants.

if both logs have the same retention and the same accesses, you can out them in the same index, otherwise you have to put them in different indexes.

In addition, it could depend on the reasons related to these logs:

do you want to make the same monitoring of the production logs?

e.g.: if you want to monitor only prod systems it's easier to have non prod logs in a different index.

Ciao.

Giuseppe

Re: Separating logs from different environments

ITWhisperer — Tue, 07 Dec 2021 09:57:22 GMT

It depends on what you want to do with the information.

Do you want to be able to distinguish which environment an event came from?

Do you want to be able to mix events from different environments into the same search/dashboard?

Can you use the host field to determine which environment the event came from, e.g. by a simple lookup?

Are the log formats the same across all environments?

Re: Separating logs from different environments

isoutamo — Tue, 07 Dec 2021 10:12:38 GMT

In the first step is looking if there are any regulation or legislations which force you to use separate environments or can you still use the same for production and test. Also you must check what kind of access restrictions there are in your enterprise for logs. Who can see production and who can access test logs. Usually those are at least partially different groups and quite often it's not allowed for any individual person to see both.

After you have gotten answers to above questions then you can continue with @gcusello's and @ITWhisperer 's guidelines.

r. Ismo

Re: Separating logs from different environments

nembela — Tue, 07 Dec 2021 11:21:26 GMT

Hi,

Thanks for the ideas. I'll try to answer the questions in one post.

@isoutamo: We have regulations that requrire the collection of logs from environments where the data is not fully anonimized (e.g. staging and test). But we can use the same Splunk instance for all these logs.

@gcusello: Because these are standard OS logs, the same teams need to access/monitor them. We don't need to give access to developers because there are no application logs.

@ITWhisperer:

Do you want to be able to distinguish which environment an event came from? Yes
Do you want to be able to mix events from different environments into the same search/dashboard? Yes
Can you use the host field to determine which environment the event came from, e.g. by a simple lookup? Yes, but it will need definitely some manual work
Are the log formats the same across all environments? Yes, normal Linux and Windows OS logs

Re: Separating logs from different environments

ITWhisperer — Tue, 07 Dec 2021 11:35:58 GMT

Possibly the simplest way is to perform a lookup at ingestion time based on the host and set an "environment" field to "tag" the event with the environment it belongs to. When the hosts for an environment change, you should just need to update the lookup store. Initial set up might need some manual work, but it provides a reasonably flexible solution should the purpose of a host moves from one environment to another as the purpose at the time of ingestion would be preserved.

Re: Separating logs from different environments

PickleRick — Tue, 07 Dec 2021 12:18:51 GMT

You can also just modify the source field. I know that typically source represents the forwarder's point of view, but sometimes it's convenient to change it. For example, when I use syslog->rsyslog->HEC infrastructure I modify the source field to include the IP of the source host. I know that sc4s adds additional field for that but we wanted to do without adding extra metadata to events.