I did not find an optimal solution for myself yet. But I guess many of you have similar use cases, so maybe you can explain how you handle such situations:
Let's say you have an application consisting of:
And you have separate environments for:
So you probably want to slice your searches different ways:
Either by server role: Show me any errors on web servers (no matter whether prod, staging or dev)
Or by server environment: Show me any errors in production environment (for all: lb, web, app, db)
So I started tagging my hosts like this:
[host=server1] production = enabled webserver = enabled [host=server2] production = enabled database = enabled [host=server3] staging = enabled webserver = enabled ...
Now I can quickly perform normal searches like
index=perfmon tag=webserver or
I know one can also use eventtypes, macros or lookups to achieve almost the same. The reason I prefer tags over the others is I can assign multiple attributes to one host easily. Doing the same with eventtypes or macros gets tedious quickly. If you add a webserver to production, you have to change two macros. And if you have more attributes (pci etc), it doesn't scale well.
So here is my current problem:
I have built a KV Store lookup storing the currently deployed application builds by host:
If I e.g. want to query the buildnumber for a specific environment I cannot use my tags:
| inputlookup my_buildnumbers where tag=webserver would be great because the filtering would be done directly in KV Store / MongoDB. But obviously this doesn't work because Mongo doesn't know my tags.
| inputlookup my_buildnumbers | eval host=_key | search tag=webserver unfortunately doesn't work either. I guess because tags aren't added for inputlookups. Only for real events.
| inputlookup my_buildnumbers | search `webservers` with the macro expanding to
(host=server1 OR host=server2) would probably work. But as mentioned I dislike the macro approach for the reason above.
Does anybody have a good idea? How do you handle such things?
thanks, this works. But depending on the search time range the subsearch can be quite slow. I think I need to find some optimizations for it.