What is your best practice for search slicing by host environment/role?

[Skorfulose]

Hey there!

I did not find an optimal solution for myself yet. But I guess many of you have similar use cases, so maybe you can explain how you handle such situations:

Let's say you have an application consisting of:

• load balancers
• web servers
• application servers
• database servers

And you have separate environments for:

• production
• staging
• development

So you probably want to slice your searches different ways:
Either by server role: Show me any errors on web servers (no matter whether prod, staging or dev)
Or by server environment: Show me any errors in production environment (for all: lb, web, app, db)

So I started tagging my hosts like this:

[host=server1]
production = enabled
webserver = enabled

[host=server2]
production = enabled
database = enabled

[host=server3]
staging = enabled
webserver = enabled
...

Now I can quickly perform normal searches like index=perfmon tag=webserver or index=perfmon tag=production.

I know one can also use eventtypes, macros or lookups to achieve almost the same. The reason I prefer tags over the others is I can assign multiple attributes to one host easily. Doing the same with eventtypes or macros gets tedious quickly. If you add a webserver to production, you have to change two macros. And if you have more attributes (pci etc), it doesn't scale well.

So here is my current problem:
I have built a KV Store lookup storing the currently deployed application builds by host: _key=host and value=buildnumber.
If I e.g. want to query the buildnumber for a specific environment I cannot use my tags:

| inputlookup my_buildnumbers where tag=webserver would be great because the filtering would be done directly in KV Store / MongoDB. But obviously this doesn't work because Mongo doesn't know my tags.

| inputlookup my_buildnumbers | eval host=_key | search tag=webserver unfortunately doesn't work either. I guess because tags aren't added for inputlookups. Only for real events.

| inputlookup my_buildnumbers | search `webservers` with the macro expanding to (host=server1 OR host=server2) would probably work. But as mentioned I dislike the macro approach for the reason above.

Does anybody have a good idea? How do you handle such things?

Cheers!
Thomas

[sundareshr · Accepted solution]

How about a sub-search

| inputlookup my_buildnumbers | search [search tag=webservers | stats count by host | table host]

[Skorfulose]

Hello sundareshr,

thanks, this works. But depending on the search time range the subsearch can be quite slow. I think I need to find some optimizations for it.

Regards,
Thomas