Knowledge Management

Separating logs from different environments

nembela
Path Finder

Hi,

 

Till now we only collected logs from production servers with Splunk. But soon we will onboard the system logs from non-prod (Linux, Windows) servers.

What is the best way to differentiate between the logs from different environents?

  • different index? All these logs have the same retention time
  • different sourcetype? All the logs are system logs (Windows, Linux)
  • eventtype?
  • a dedicated "environment" field?
  • tagging?

Thanks,

Laci

Labels (3)
0 Karma
1 Solution

ITWhisperer
SplunkTrust
SplunkTrust

Possibly the simplest way is to perform a lookup at ingestion time based on the host and set an "environment" field to "tag" the event with the environment it belongs to. When the hosts for an environment change, you should just need to update the lookup store. Initial set up might need some manual work, but it provides a reasonably flexible solution should the purpose of a host moves from one environment to another as the purpose at the time of ingestion would be preserved.

View solution in original post

0 Karma

nembela
Path Finder

Hi,

 

Thanks for the ideas. I'll try to answer the questions in one post.

@isoutamo: We have regulations that requrire the collection of logs from environments where the data is not fully anonimized (e.g. staging and test). But we can use the same Splunk instance for all these logs.

@gcusello: Because these are standard OS logs, the same teams need to access/monitor them. We don't need to give access to developers because there are no application logs.

@ITWhisperer:

  • Do you want to be able to distinguish which environment an event came from? Yes
  • Do you want to be able to mix events from different environments into the same search/dashboard? Yes
  • Can you use the host field to determine which environment the event came from, e.g. by a simple lookup? Yes, but it will need definitely some manual work
  • Are the log formats the same across all environments? Yes, normal Linux and Windows OS logs
0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Possibly the simplest way is to perform a lookup at ingestion time based on the host and set an "environment" field to "tag" the event with the environment it belongs to. When the hosts for an environment change, you should just need to update the lookup store. Initial set up might need some manual work, but it provides a reasonably flexible solution should the purpose of a host moves from one environment to another as the purpose at the time of ingestion would be preserved.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

You can also just modify the source field. I know that typically source represents the forwarder's point of view, but sometimes it's convenient to change it. For example, when I use syslog->rsyslog->HEC infrastructure I modify the source field to include the IP of the source host. I know that sc4s adds additional field for that but we wanted to do without adding extra metadata to events.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

It depends on what you want to do with the information.

Do you want to be able to distinguish which environment an event came from?

Do you want to be able to mix events from different environments into the same search/dashboard?

Can you use the host field to determine which environment the event came from, e.g. by a simple lookup?

Are the log formats the same across all environments?

isoutamo
SplunkTrust
SplunkTrust

Hi

In the first step is looking if there are any regulation or legislations which force you to use separate environments or can you still use the same for production and test. Also you must check what kind of access restrictions there are in your enterprise for logs. Who can see production and who can access test logs. Usually those are at least partially different groups and quite often it's not allowed for any individual person to see both.

After you have gotten answers to above questions then you can continue with @gcusello's and @ITWhisperer 's guidelines.

r. Ismo

gcusello
SplunkTrust
SplunkTrust

Hi @nembela,

at first the choose of an index depends on two reasons.

  • retention,
  • accesses.

usually non prod logs have a different retention and different access grants.

if both logs have the same retention and the same accesses, you can out them in the same index, otherwise you have to put them in different indexes.

In addition, it could depend on the reasons related to these logs:

do you want to make the same monitoring of the production logs?

e.g.: if you want to monitor only prod systems it's easier to have non prod logs in a different index.

Ciao.

Giuseppe

Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...