How rex field list values assign dynamically to source path as subquery ?

[alok]

Hello,

Query one returns a result with one fields as list of values. I want to  pass those list of value as the search source path and result returns for second query. Given below is the detail.

Please suggest how to achieve ? 

Query1 : 

index="os" (source="/var/log/steps/*/controller")  sourcetype="too_small" (host="ip-101-108-*-*" OR host="ip-101-109-*-*") | transaction source startswith=("/code/ttt_env.sh" OR "/code/ttt_gen.sh" ) endswith="startRun() called" | rex field=_raw "(?<step_function>\bs-[a-zA-Z0-9_]+)"

It does return the output and value of 

Query1 Output : 

step_function values listed as  in field like : s-BBBUL8NJBYE45, s-AAAUL8NJBYEI3

Now these value I want to generate the further query using step_function values like ( Hard coded by hand it worked)

append [search index="os" source=("/var/log/steps/s-BBBUL8NJBYE45/stdout" OR /var/log/steps/s-s-AAAUL8NJBYEI3/stdout")  sourcetype="too_small" (host="ip-101-108-*-*"" OR host="ip-101-108-*-*"*")]

How to perform dynamically and achieve this functionality without hardcoding. 

Tried like this but didn't work 

index="os" (source="/var/log/steps/*/controller") sourcetype="too_small" (host="ip-101-108-*-*" OR host="ip-101-108-*-*") |
transaction source startswith=("/code/ttt_env.sh" OR "/code/ttt_gen.sh") endswith="startRun() called" |
rex field=_raw "(?<rec_prod_step_function>\bs-[a-zA-Z0-9_]+)" | search rec_prod_step_function="*"
| append [search index="os" source="/var/log/steps/$rec_prod_step_function$/stdout" sourcetype="too_small" (host="ip-101-108-*-*" OR host="ip-101-108-*-*")]

Note : "/var/log/steps/$rec_prod_step_function$/stdout"

Thanks in advance.

[ITWhisperer]

Rather than append, try using map

index="os" (source="/var/log/steps/*/controller") sourcetype="too_small" (host="ip-101-108-*-*" OR host="ip-101-108-*-*") |
transaction source startswith=("/code/ttt_env.sh" OR "/code/ttt_gen.sh") endswith="startRun() called" |
rex field=_raw "(?<rec_prod_step_function>\bs-[a-zA-Z0-9_]+)" | where rec_prod_step_function="*"
| map search="search index=\"os\" source=\"/var/log/steps/$rec_prod_step_function$/stdout\" sourcetype=\"too_small\" (host=\"ip-101-108-*-*\" OR host=\"ip-101-108-*-*\")" maxsearches=0

[alok]

I ran the suggested query getting a error message 

Error in 'map': Did not find value for required attribute 'rec_prod_step_function'.

Please suggest.

As debug I break the query when I ran 

index="os" (source="/var/log/steps/*/controller") sourcetype="too_small" (host="ip-101-108-*-*" OR host="ip-101-108-*-*") |
transaction source startswith=("/code/ttt_env.sh" OR "/code/ttt_gen.sh") endswith="startRun() called" |
rex field=_raw "(?<rec_prod_step_function>\bs-[a-zA-Z0-9_]+)" | where rec_prod_step_function="*"

It is not returning any event.

but when I used "where" to "search"

index="os" (source="/var/log/steps/*/controller") sourcetype="too_small" (host="ip-101-108-*-*" OR host="ip-101-108-*-*") |
transaction source startswith=("/code/ttt_env.sh" OR "/code/ttt_gen.sh") endswith="startRun() called" |
rex field=_raw "(?<rec_prod_step_function>\bs-[a-zA-Z0-9_]+)" | search rec_prod_step_function="*"

Query returns two events that is correct.

Please suggest.

Thanks !!

[ITWhisperer]

If you get the right results with search instead of where, does the map function do what you want?

I don't understand why search works but where doesn't. Does the rec_prod_step_function field get extracted successfully? Can you provide the results of the successful query?