Searching across multiple host combinations

[larrydavid]

Our team looks after 7 applications, we have 5 environments and each application sits on between 2 and 4 servers, depending on the environment.

Each app instance has its own dedicated server, so in other words, given a hostname, you can figure out exactly which application and which environment it is for.

At the moment, if we want to search for the logs of one of the applications (app1) in UAT, and if this app has 4 servers in UAT, the only way we can do this is by using the following search parameters

source=*app1.log host=host1 OR host=host2 OR host=host3 OR host=host4

Sometimes we have a few different applications talking to each other, so we end up having to mention a long list of host names and this gets quite tedious. 

We have a separate team that manages Splunk across the organisation.

Is there something we could be asking the Splunk team to do for us to make our searching easier?

Is there something they could do that would result in us being able to do something like

application=app1 environment=uat 

instead of having to specify host names for the environment that we are interested in?

Our team would appreciate any suggestions that can make our work easier.

Thank you

[gcusello]

Hi @larrydavid ,

the easiest approach is to create a lookup (eventually an automatic one!) containing the combinations of apps and hosts to define the environments, so you can use the lookup in your searches, something like this:

environment app  host
env1        app1 host1
env1        app1 host2
env1        app1 host3
env2        app2 host4
env2        app2 host5
env2        app2 host6
env3        app3 host7
env3        app3 host8
env3        app3 host9

One additiona question: if each application uses some servers and there's a relation 1:n between apps and hosts, why you don't use only apps to define your environment?

then, remember the there's the IN() operator to use instead of OR:

source=*app1.log host IN (host1,host2,host3,host4)

it's smaller!

Ciao.

Giuseppe 

[yuanliu]

You need to clarify your constraints.  The most obvious solution is to send a field "environment" along with log events.  There are a million ways to do this.

Then, if the deployment team is sympathetic to your course, they can name hosts according to environment in some way.  There is at least a dozen ways to do this. (One obvious way is to dedicate a special domain to environment.)  So, that's at least 1,000,012.

You can also do an automatic lookup on hostname.  That's at least 1,000,013 ways.