How to distinguish applications log by environment?

[GaetanVP]

Hello Splunkers,

Here is my use-case : I am monitoring apache logs on 3 different VMs, one VM for each env : dev, uat, prod

I do not see the point to create a specific index for each env (no security / restrictions needed). But I still want to be able to distinguish the logs by environment. 

What would be the best practice to do that ? Create a tag / event type ? for each host from where the logs are coming ?

Regards,
GaetanVP

[PickleRick · Accepted solution]

It depends on your case.

If you don't need to restrict access to particular dev (like you have a team which should see logs from dev but not from prod) and don't need to apply different retention policies you generally don't need to split the events into different indexes. (OK, if your dev or uat generated way way less events than prod, that could also be the case for dedicating a separate index for it).

Assuming that you don't split it into different indexes, question is whether your data is distinguishable by any part of data/metadata. If - for example - your host field is different, you're all set. You know how to search for data from prod (by limiting your search with "host IN (hosta,hostb, hostc)". To make search more convenient you can create an eventtype or alias (typically alias is often used to externalize a configurable part from searches so that you can make many static searches - for example in a dashboard - and you can easily change set of data involved used by just redefining the macro).

If you don't have any metadata field which can help you tell the source environment, you have to generate one by means of additional indexed field.

[scelikok]

Hi @GaetanVP,

Actually, you do not need anything specific to distinguish. They are already different hosts. You can easily search using the host like below;

index=apache host=dev

  Or you can check host field to distinguish the environment

[tej57]

Hello @GaetanVP,

You definitely do not need to create 3 different indexes for each environment. However based on the host, you can route them to a specific sourcetype by using below props.conf.

props.conf

[host::<<dev_host>>]
sourcetype = <<orig_sourcetype:dev>>

[host::<<uat_host>>]
sourcetype = <<orig_sourcetype:uat>>

[host::<<prod_host>>]
sourcetype = <<orig_sourcetype:prod>>

This will ingest all the data into one index and whenever you wish to search the logs for specific environment, you can filter the logs based on the sourcetype. Additionally, if you expand the number of hosts in any environment, you can provide the regex pattern for hostnames to route them to env specific sourcetype.

[PickleRick]

No!

Don't touch the sourcetype!

Sourcetype is the most basic level of saying "this format of events contains this and that" so if you change the sourcetype you'd have to adjust a whole bunch of other configuration items. So no, sourcetype is not the way to go.

Not to mention that your syntax won't work anyway.