I did not find an optimal solution for myself yet. But I guess many of you have similar use cases, so maybe you can explain how you handle such situations:
Let's say you have an application consisting of:
And you have separate environments for:
So you probably want to slice your searches different ways:
Either by server role: Show me any errors on web servers (no matter whether prod, staging or dev)
Or by server environment: Show me any errors in production environment (for all: lb, web, app, db)
Now I can quickly perform normal searches like index=perfmon tag=webserver or index=perfmon tag=production.
I know one can also use eventtypes, macros or lookups to achieve almost the same. The reason I prefer tags over the others is I can assign multiple attributes to one host easily. Doing the same with eventtypes or macros gets tedious quickly. If you add a webserver to production, you have to change two macros. And if you have more attributes (pci etc), it doesn't scale well.
So here is my current problem:
I have built a KV Store lookup storing the currently deployed application builds by host: _key=host and value=buildnumber.
If I e.g. want to query the buildnumber for a specific environment I cannot use my tags:
| inputlookup my_buildnumbers where tag=webserver would be great because the filtering would be done directly in KV Store / MongoDB. But obviously this doesn't work because Mongo doesn't know my tags.
| inputlookup my_buildnumbers | eval host=_key | search tag=webserver unfortunately doesn't work either. I guess because tags aren't added for inputlookups. Only for real events.
| inputlookup my_buildnumbers | search `webservers` with the macro expanding to (host=server1 OR host=server2) would probably work. But as mentioned I dislike the macro approach for the reason above.
Does anybody have a good idea? How do you handle such things?