After talking to our technical account manager, this is the solution we came up with, it works very well:
We run a scheduled search every five minutes on the search head, that gets the complete list of deployment clients from our deployment server (replace deploymentsever with the host name of your deployment server, or use local if it's the same as your search head) and then stores them in a static lookup file.
| rest /services/deployment/server/clients count=0 splunk_server=deploymentserver | fields hostname name | rename name as clientName | outputlookup clientNames.csv
Then we add the below stanzas to props.conf and transforms.conf so all messages get their clientName looked up based on their host:
props.conf:
[host::*]
LOOKUP-client = clientLookup hostname AS host OUTPUT clientName
transforms.conf:
[clientLookup]
filename = clientNames.csv
And now, we can simply type clientName=new_web_app into the Search app, and all of our logs from that app will come up. Works great, and haven't seen any noticeable performance hit from the extra lookups.
... View more