Suppose I performed the following subsearch
index=whatever "name=" [|inputlookup lookup_file.csv | return 100 $lookup_id]
And lookup_file.csv has only id=456. Against thousands of logs, but one of those log events has the following output:
[name="ABC", id=123, name="DEF", id=456]
I saw that when I performed a search similar to this, the log would be returned with the lookup_id of 456 even though both 123 and 456 were present in the log. Is it expected behavior for inputlookup to return this log even if id=123 is found before id=456? Basically, does inputlookup return logs that have multiple values for the same field?
Your event has BOTH id=123
AND id=456
so the result is correct and the results of all of these are the same for this event:
... id=123
... id=456
... id=123 OR id=456
... id=123 AND id=456
This makes perfect sense; to alter, you could add this:
... | where mvcount(id) == 1
You need to first understand what your inputlookup subsearch is doing here.
Your original search:
index=whatever "name=" [|inputlookup lookup_file.csv | return 100 $lookup_id]
The subsearch will collect first 100 values of field lookup_id in lookup_file.csv and create giant OR condition statement with just the value of the field.
e.g. ( ("lookup_id1") OR ("lookup_id2") OR....("lookup_id100"))
Your search will basically be (internally) tranformed like this
index=whatever "name=" ( ("lookup_id1") OR ("lookup_id2") OR....("lookup_id100"))
So you'll basically be doing a text search with those Id values and it'll return all records where that string is present. In your sample log [name="ABC", id=123, name="DEF", id=456]
, value 456
is present, hence it'll be returned. It's a plain text search so it'll not do any additional checks like only give where id=456 and no other values
.
Awesome, thanks for that! I thought there was extra logic in place to check no other values. Is it safe to say that since it's a plain text search, any record can be returned if it even partially matches the sequence of characters in the lookup table (i.e. id=4567
will cause the record to be returned)?