- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Search result not in second with rex fields
I want to look for requests in a service mesh ingest log which have no corresponding application log entries.
My first search is
index=kubernetes source=*envoy-proxy* (api.foo.com OR info) AND downstream_remote_disconnect | rex field=_raw "\[[^\]]+\] \"(?<downstream>[^\"]+)\".*\"(POST|GET) \"(?<host>[^\"]+)\" \"(?<path>[^\"\?]+)[\?]?\" [^\"]+\" (?<status>\d+).*\"(?<id1>[0-9a-f]{8})-(?<id2>[0-9a-f]{4})-(?<id3>[0-9a-f]{4})" | eval id=id1.id2.id3 | fields id
my second search is
index=kubernetes source=*proxy* operation: | rex field=_raw "span_id:(?<id>[0-9a-f]{16});" | fields id
and the obvious way of combining them yields no results
index=kubernetes source=*envoy-proxy* (api.foo.com OR info) AND downstream_remote_disconnect | rex field=_raw "\[[^\]]+\] \"(?<downstream>[^\"]+)\".*\"(POST|GET) \"(?<host>[^\"]+)\" \"(?<path>[^\"\?]+)[\?]?\" [^\"]+\" (?<status>\d+).*\"(?<id1>[0-9a-f]{8})-(?<id2>[0-9a-f]{4})-(?<id3>[0-9a-f]{4})" | eval id=id1.id2.id3 | fields id | search NOT [ search index=kubernetes source=*proxy* operation: | rex field=_raw "span_id:(?<id>[0-9a-f]{16});" | fields id ]
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content


Do the two searches work independently? Do they produce id field values that match?
If the two searches produce the same set of IDs then they'll cancel each other out and you'll get no results.
If this reply helps you, Karma would be appreciated.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The searches work independently.
The first search has events with id fields that do not exist in the second search.
I learned this when I manually went through 50+ entries from the first search and looked for them with an AND clause in the second search producing no matches.
I'd like to automate that process.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content


Have you tried formatting the results of the second search?
index=kubernetes source=*envoy-proxy* (api.foo.com OR info) AND downstream_remote_disconnect
| rex field=_raw "\[[^\]]+\] \"(?<downstream>[^\"]+)\".*\"(POST|GET) \"(?<host>[^\"]+)\" \"(?<path>[^\"\?]+)[\?]?\" [^\"]+\" (?<status>\d+).*\"(?<id1>[0-9a-f]{8})-(?<id2>[0-9a-f]{4})-(?<id3>[0-9a-f]{4})"
| eval id=id1.id2.id3
| fields id
| search NOT [
search index=kubernetes source=*proxy* operation:
| rex field=_raw "span_id:(?<id>[0-9a-f]{16});"
| fields id
| format
]
If this reply helps you, Karma would be appreciated.
