On our forwarders we have a [default] _meta value that specify a few key::value pairs, e.g. a key to tell us what site this server forwarding the data belongs to (e.g. site::staging ).
When I search for site=staging I get fewer results than with site::staging , and I can't seem to find relevant documentation that explains why and it isn't easy to determine what is actually being filtered in the first compared to the latter when I look at the results (no easy/logical at-a-glance answer).
If I take a specific host and a simple file we monitor, like host=someserver source=/var/log/messages - the result set is vastly different, with site::staging containing what seems to be the entire log and site=staging giving just a few lines from the log (and those lines don't contain text which mention staging either). I can't think of any logic that would dictate why just a few lines seem to match site=staging - I'd expect that field to either be searchable or not, not some odd subset of data. Also the UI highlights the source field in the events as matching staging when I do site::staging which points to it thinking that it knows what I want. This makes it even more annoying when using the UI to add a field to the search, because when you click the site field and "add to search" it adds it as site=staging which doesn't yield all the results.
This is with SplunkCloud if that makes any difference.
... View more