Hi,
Till now we only collected logs from production servers with Splunk. But soon we will onboard the system logs from non-prod (Linux, Windows) servers.
What is the best way to differentiate between the logs from different environents?
Thanks,
Laci
Possibly the simplest way is to perform a lookup at ingestion time based on the host and set an "environment" field to "tag" the event with the environment it belongs to. When the hosts for an environment change, you should just need to update the lookup store. Initial set up might need some manual work, but it provides a reasonably flexible solution should the purpose of a host moves from one environment to another as the purpose at the time of ingestion would be preserved.
Hi,
Thanks for the ideas. I'll try to answer the questions in one post.
@isoutamo: We have regulations that requrire the collection of logs from environments where the data is not fully anonimized (e.g. staging and test). But we can use the same Splunk instance for all these logs.
@gcusello: Because these are standard OS logs, the same teams need to access/monitor them. We don't need to give access to developers because there are no application logs.
Possibly the simplest way is to perform a lookup at ingestion time based on the host and set an "environment" field to "tag" the event with the environment it belongs to. When the hosts for an environment change, you should just need to update the lookup store. Initial set up might need some manual work, but it provides a reasonably flexible solution should the purpose of a host moves from one environment to another as the purpose at the time of ingestion would be preserved.
You can also just modify the source field. I know that typically source represents the forwarder's point of view, but sometimes it's convenient to change it. For example, when I use syslog->rsyslog->HEC infrastructure I modify the source field to include the IP of the source host. I know that sc4s adds additional field for that but we wanted to do without adding extra metadata to events.
It depends on what you want to do with the information.
Do you want to be able to distinguish which environment an event came from?
Do you want to be able to mix events from different environments into the same search/dashboard?
Can you use the host field to determine which environment the event came from, e.g. by a simple lookup?
Are the log formats the same across all environments?
Hi
In the first step is looking if there are any regulation or legislations which force you to use separate environments or can you still use the same for production and test. Also you must check what kind of access restrictions there are in your enterprise for logs. Who can see production and who can access test logs. Usually those are at least partially different groups and quite often it's not allowed for any individual person to see both.
After you have gotten answers to above questions then you can continue with @gcusello's and @ITWhisperer 's guidelines.
r. Ismo
Hi @nembela,
at first the choose of an index depends on two reasons.
usually non prod logs have a different retention and different access grants.
if both logs have the same retention and the same accesses, you can out them in the same index, otherwise you have to put them in different indexes.
In addition, it could depend on the reasons related to these logs:
do you want to make the same monitoring of the production logs?
e.g.: if you want to monitor only prod systems it's easier to have non prod logs in a different index.
Ciao.
Giuseppe