What's Wrong With This Query?

[ahcarpenter]

Hoping to filter a search based on a list of values from a subquery where in both cases it's matching against a rex'd field.

index=x 
  [ search index=x 2e5b422130e64645cb9681a32fd28cb6
     | rex "downstreamTraceID\=\{ (?<downstream_trace_id>.{32})"
     | fields downstream_trace_id
  ]
 | rex "downstreamTraceID\=\{ (?<downstream_trace_id>.{32})"

[bowesmana]

@ahcarpenter 

I am assuming that the field you are creating in your subsearch 'downstream_trace_id', is not already an auto extracted field in the event, so in that case, this will not work, as you are asking the outer search (which runs AFTER the subsearch) to filter on a field that does not yet exist.

Also, with the subsearch, you would be sensible in any case to do a 

| stats values(downstream_trace_id) as downstream_trace_id

I am not sure I understand the need for the subsearch anyway, as presumably this would give you the result you are after

index=x 2e5b422130e64645cb9681a32fd28cb6
| rex "downstreamTraceID\=\{ (?<downstream_trace_id>.{32})"

If I understand that the query is looking for all events containing '2e5b422130e64645cb9681a32fd28cb6' and then trying to collect all the downstream ids for that primary id

If you do want/need to use subsearches and that downstream_trace_id field is one that you would use frequently, then it's probably worth setting up a field extraction, so that it is available as part of the search, so you don't need to always run the rex statement.

Hope this helps

[ahcarpenter]

Thanks a ton for the response @bowesmana !

You're right in assuming we can't use field extraction atm (short of it).

It's a little confusing, but the subsearch is to get all downstream trace-ids associated with an upstream trace-id (there's a trace-id field and also a dowstreamTraceID field)

Hoping to take the downstream values and then filter against an unfiltered search as there can be logs with Trace-Ids that contain a matching downstream id but not matching Trace-Id

Any thoughts on the best way to handle that without field-extraction?

[bowesmana]

@ahcarpenter 

so if I understand correctly, you're looking to get all downstream ids associated with trace-id=2e5b422130e64645cb9681a32fd28cb6 and then find all events that have any of those downstream ids.

So, you can use the where clause and the subsearch, which allows you to do the field extraction before the where, so the downstream id field will then be visible in the where clause from the results of the subsearch.

index=x 
| rex "downstreamTraceID\=\{ (?<downstream_trace_id>.{32})"
| where [ search index=x 2e5b422130e64645cb9681a32fd28cb6
     | rex "downstreamTraceID\=\{ (?<downstream_trace_id>.{32})"
     | stats values(downstream_trace_id) as downstream_trace_id
  ]

There are probably other ways to achieve the same thing, but I think this should work.