Perhaps something like
| inputlookup servers.csv where NOT [|inputlookup HR.csv | format]
Using subsearch results in large number of OR operators. It's probably more economic just doing stats
| inputlookup servers.csv
| eval CSV = "servers"
| inputlookup append=true HR.csv
| fillnull CSV value=HR
| stats values(CSV) as CSV by Name ID
| where mvcount(CSV) == 1 AND CSV == "servers"
(Again, thanks @richgalloway for demonstrating append mode!)