Search result not in second with rex fields

[drew_eckhardt]

I want to look for requests in a service mesh ingest log which have no corresponding application log entries.

My first search is 

index=kubernetes source=*envoy-proxy*  (api.foo.com OR info) AND downstream_remote_disconnect 
| rex field=_raw "\[[^\]]+\] \"(?<downstream>[^\"]+)\".*\"(POST|GET) \"(?<host>[^\"]+)\" \"(?<path>[^\"\?]+)[\?]?\" [^\"]+\" (?<status>\d+).*\"(?<id1>[0-9a-f]{8})-(?<id2>[0-9a-f]{4})-(?<id3>[0-9a-f]{4})"
| eval id=id1.id2.id3
| fields id

my second search is 

index=kubernetes source=*proxy* operation:
| rex field=_raw "span_id:(?<id>[0-9a-f]{16});"
| fields id

and the obvious way of combining them yields no results

index=kubernetes source=*envoy-proxy*  (api.foo.com OR info) AND downstream_remote_disconnect 
| rex field=_raw "\[[^\]]+\] \"(?<downstream>[^\"]+)\".*\"(POST|GET) \"(?<host>[^\"]+)\" \"(?<path>[^\"\?]+)[\?]?\" [^\"]+\" (?<status>\d+).*\"(?<id1>[0-9a-f]{8})-(?<id2>[0-9a-f]{4})-(?<id3>[0-9a-f]{4})"
| eval id=id1.id2.id3
| fields id
| search NOT [
search index=kubernetes source=*proxy* operation:
| rex field=_raw "span_id:(?<id>[0-9a-f]{16});"
| fields id
]

[richgalloway]

Do the two searches work independently?  Do they produce id field values that match?

If the two searches produce the same set of IDs then they'll cancel each other out and you'll get no results.

[drew_eckhardt]

The searches work independently.

The first search has events with id fields that do not exist in the second search.

I learned this when I manually went through 50+ entries from the first search and looked for them with an AND clause in the second search producing no matches.

I'd like to automate that process.

[richgalloway]

Have you tried formatting the results of the second search?

index=kubernetes source=*envoy-proxy*  (api.foo.com OR info) AND downstream_remote_disconnect 
| rex field=_raw "\[[^\]]+\] \"(?<downstream>[^\"]+)\".*\"(POST|GET) \"(?<host>[^\"]+)\" \"(?<path>[^\"\?]+)[\?]?\" [^\"]+\" (?<status>\d+).*\"(?<id1>[0-9a-f]{8})-(?<id2>[0-9a-f]{4})-(?<id3>[0-9a-f]{4})"
| eval id=id1.id2.id3
| fields id
| search NOT [
  search index=kubernetes source=*proxy* operation:
  | rex field=_raw "span_id:(?<id>[0-9a-f]{16});"
  | fields id
  | format
  ]