And lookup_file.csv has only id=456. Against thousands of logs, but one of those log events has the following output:
[name="ABC", id=123, name="DEF", id=456]
I saw that when I performed a search similar to this, the log would be returned with the lookup_id of 456 even though both 123 and 456 were present in the log. Is it expected behavior for inputlookup to return this log even if id=123 is found before id=456? Basically, does inputlookup return logs that have multiple values for the same field?
The subsearch will collect first 100 values of field lookup_id in lookup_file.csv and create giant OR condition statement with just the value of the field.
e.g. ( ("lookup_id1") OR ("lookup_id2") OR....("lookup_id100"))
Your search will basically be (internally) tranformed like this
index=whatever "name=" ( ("lookup_id1") OR ("lookup_id2") OR....("lookup_id100"))
So you'll basically be doing a text search with those Id values and it'll return all records where that string is present. In your sample log [name="ABC", id=123, name="DEF", id=456], value 456 is present, hence it'll be returned. It's a plain text search so it'll not do any additional checks like only give where id=456 and no other values.
Awesome, thanks for that! I thought there was extra logic in place to check no other values. Is it safe to say that since it's a plain text search, any record can be returned if it even partially matches the sequence of characters in the lookup table (i.e. id=4567 will cause the record to be returned)?