[inputlookup] Does inputlookup return logs that have multiple values for the same field?

[aakines]

Suppose I performed the following subsearch

index=whatever "name=" [|inputlookup lookup_file.csv | return 100 $lookup_id]

And lookup_file.csv has only id=456. Against thousands of logs, but one of those log events has the following output:

[name="ABC", id=123, name="DEF", id=456]

I saw that when I performed a search similar to this, the log would be returned with the lookup_id of 456 even though both 123 and 456 were present in the log. Is it expected behavior for inputlookup to return this log even if id=123 is found before id=456? Basically, does inputlookup return logs that have multiple values for the same field?

[woodcock]

Your event has BOTH id=123 AND id=456 so the result is correct and the results of all of these are the same for this event:

... id=123
... id=456
... id=123 OR id=456
... id=123 AND id=456

This makes perfect sense; to alter, you could add this:

... | where mvcount(id) == 1

[somesoni2]

You need to first understand what your inputlookup subsearch is doing here.

Your original search:

index=whatever "name=" [|inputlookup lookup_file.csv | return 100 $lookup_id]

The subsearch will collect first 100 values of field lookup_id in lookup_file.csv and create giant OR condition statement with just the value of the field.

e.g. ( ("lookup_id1") OR ("lookup_id2") OR....("lookup_id100"))

Your search will basically be (internally) tranformed like this

 index=whatever "name=" ( ("lookup_id1") OR ("lookup_id2") OR....("lookup_id100"))

So you'll basically be doing a text search with those Id values and it'll return all records where that string is present. In your sample log [name="ABC", id=123, name="DEF", id=456], value 456 is present, hence it'll be returned. It's a plain text search so it'll not do any additional checks like only give where id=456 and no other values.

[aakines]

Awesome, thanks for that! I thought there was extra logic in place to check no other values. Is it safe to say that since it's a plain text search, any record can be returned if it even partially matches the sequence of characters in the lookup table (i.e. id=4567 will cause the record to be returned)?