Splunk Search

Search result not in second with rex fields

drew_eckhardt
Engager

I want to look for requests in a service mesh ingest log which have no corresponding application log entries.

My first search is 

index=kubernetes source=*envoy-proxy*  (api.foo.com OR info) AND downstream_remote_disconnect 
| rex field=_raw "\[[^\]]+\] \"(?<downstream>[^\"]+)\".*\"(POST|GET) \"(?<host>[^\"]+)\" \"(?<path>[^\"\?]+)[\?]?\" [^\"]+\" (?<status>\d+).*\"(?<id1>[0-9a-f]{8})-(?<id2>[0-9a-f]{4})-(?<id3>[0-9a-f]{4})"
| eval id=id1.id2.id3
| fields id

my second search is 

index=kubernetes source=*proxy* operation:
| rex field=_raw "span_id:(?<id>[0-9a-f]{16});"
| fields id

and the obvious way of combining them yields no results

index=kubernetes source=*envoy-proxy*  (api.foo.com OR info) AND downstream_remote_disconnect 
| rex field=_raw "\[[^\]]+\] \"(?<downstream>[^\"]+)\".*\"(POST|GET) \"(?<host>[^\"]+)\" \"(?<path>[^\"\?]+)[\?]?\" [^\"]+\" (?<status>\d+).*\"(?<id1>[0-9a-f]{8})-(?<id2>[0-9a-f]{4})-(?<id3>[0-9a-f]{4})"
| eval id=id1.id2.id3
| fields id
| search NOT [
search index=kubernetes source=*proxy* operation:
| rex field=_raw "span_id:(?<id>[0-9a-f]{16});"
| fields id
]
Labels (3)

richgalloway
SplunkTrust
SplunkTrust

Do the two searches work independently?  Do they produce id field values that match?

If the two searches produce the same set of IDs then they'll cancel each other out and you'll get no results.

---
If this reply helps you, Karma would be appreciated.
0 Karma

drew_eckhardt
Engager

The searches work independently.

The first search has events with id fields that do not exist in the second search.

I learned this when I manually went through 50+ entries from the first search and looked for them with an AND clause in the second search producing no matches.

I'd like to automate that process.

Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Have you tried formatting the results of the second search?

index=kubernetes source=*envoy-proxy*  (api.foo.com OR info) AND downstream_remote_disconnect 
| rex field=_raw "\[[^\]]+\] \"(?<downstream>[^\"]+)\".*\"(POST|GET) \"(?<host>[^\"]+)\" \"(?<path>[^\"\?]+)[\?]?\" [^\"]+\" (?<status>\d+).*\"(?<id1>[0-9a-f]{8})-(?<id2>[0-9a-f]{4})-(?<id3>[0-9a-f]{4})"
| eval id=id1.id2.id3
| fields id
| search NOT [
  search index=kubernetes source=*proxy* operation:
  | rex field=_raw "span_id:(?<id>[0-9a-f]{16});"
  | fields id
  | format
  ]
---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Buttercup Games: Further Dashboarding Techniques

Hello! We are excited to kick off a new series of blogs from SplunkTrust member ITWhisperer, who demonstrates ...

Message Parsing in SOCK

Introduction This blog post is part of an ongoing series on SOCK enablement. In this blog post, I will write ...

Exploring the OpenTelemetry Collector’s Kubernetes annotation-based discovery

We’ve already explored a few topics around observability in a Kubernetes environment -- Common Failures in a ...