topic Re: Search result not in second with rex fields in Splunk Search

Search result not in second with rex fields

drew_eckhardt — Wed, 29 Dec 2021 19:49:03 GMT

I want to look for requests in a service mesh ingest log which have no corresponding application log entries.

My first search is

index=kubernetes source=*envoy-proxy*  (api.foo.com OR info) AND downstream_remote_disconnect 
| rex field=_raw "\[[^\]]+\] \"(?<downstream>[^\"]+)\".*\"(POST|GET) \"(?<host>[^\"]+)\" \"(?<path>[^\"\?]+)[\?]?\" [^\"]+\" (?<status>\d+).*\"(?<id1>[0-9a-f]{8})-(?<id2>[0-9a-f]{4})-(?<id3>[0-9a-f]{4})"
| eval id=id1.id2.id3
| fields id

my second search is

index=kubernetes source=*proxy* operation:
| rex field=_raw "span_id:(?<id>[0-9a-f]{16});"
| fields id

and the obvious way of combining them yields no results

index=kubernetes source=*envoy-proxy*  (api.foo.com OR info) AND downstream_remote_disconnect 
| rex field=_raw "\[[^\]]+\] \"(?<downstream>[^\"]+)\".*\"(POST|GET) \"(?<host>[^\"]+)\" \"(?<path>[^\"\?]+)[\?]?\" [^\"]+\" (?<status>\d+).*\"(?<id1>[0-9a-f]{8})-(?<id2>[0-9a-f]{4})-(?<id3>[0-9a-f]{4})"
| eval id=id1.id2.id3
| fields id
| search NOT [
search index=kubernetes source=*proxy* operation:
| rex field=_raw "span_id:(?<id>[0-9a-f]{16});"
| fields id
]

Re: Search result not in second with rex fields

richgalloway — Wed, 29 Dec 2021 19:34:03 GMT

Do the two searches work independently? Do they produce id field values that match?

If the two searches produce the same set of IDs then they'll cancel each other out and you'll get no results.

Re: Search result not in second with rex fields

drew_eckhardt — Wed, 29 Dec 2021 19:46:54 GMT

The searches work independently.

The first search has events with id fields that do not exist in the second search.

I learned this when I manually went through 50+ entries from the first search and looked for them with an AND clause in the second search producing no matches.

I'd like to automate that process.

Re: Search result not in second with rex fields

richgalloway — Wed, 29 Dec 2021 21:11:02 GMT

Have you tried formatting the results of the second search?

index=kubernetes source=*envoy-proxy* (api.foo.com OR info) AND downstream_remote_disconnect | rex field=_raw "\[[^\]]+\] \"(?<downstream>[^\"]+)\".*\"(POST|GET) \"(?<host>[^\"]+)\" \"(?<path>[^\"\?]+)[\?]?\" [^\"]+\" (?<status>\d+).*\"(?<id1>[0-9a-f]{8})-(?<id2>[0-9a-f]{4})-(?<id3>[0-9a-f]{4})" | eval id=id1.id2.id3 | fields id | search NOT [ search index=kubernetes source=*proxy* operation: | rex field=_raw "span_id:(?<id>[0-9a-f]{16});" | fields id | format ]