I figured out that using a subsearch ie [ ] is not ideal in most situations. Its better to use the map search="" notation and escape the quotes inside the quoted search=.
Thus, this search works:
index=log sourcetype=app_log "keyword" | rex "(?i)primary key: (?P [^ ]+)" | join type=outer host [ | inputlookup db_info.csv ] | dedup host, primary_key | fields SID, primary_key | map search="| dbquery $SID$ \"select column1, column2 from $SID$.table where _id = '$primary_key$'\"" | table column1, column2
I also noticed that if you remove the last "table" command, and run it straight out, you wont see the result but Splunk displays results count.
... View more