Using subsearch fields in Main Search

[RobKelley06]

I am trying to do a search to get all of the POID values and then use them in a 2nd search to see if they appear anywhere.  How can I accomplish this?

index=aws_esf_prod sourcetype="aws:/aws/lambda/lambda-TS4-ESF-transmit-security-activity*" ("journeyId*Q&A" OR "journeyId*OTP Journey" OR "journeyId*fido2_auth" OR "journeyId*mobile_approve") "'policy_version_id'" "device_model" "journeyId" "'poid': '" + POID + "'"
[
  search index=aws_esf_prod sourcetype=aws:ecs source="*sef/sef-app*" "*Detected Fraud IP*" AND NOT "*56789*" AND NOT "*1234*"
  | rex field=_raw "Detected Fraud IP for the User with POID: (?<POID>\d+) and IP Address : (?<IP>.+) and Risk Score : (?<IPRISKSCORE>\d+) and Risk Type : (?<RISKTYPE>\D+)Evidence Details : (?<EVIDENCEDETAILS>.+)"
  | table POID, IP, IPRISKSCORE, RISKTYPE, EVIDENCEDETAILS
]
| timechart span=1m cont=false 
values(POID) AS POID
values(IP) AS IP_ADDRESS 
values(IPRISKSCORE) AS RISK_SCORE
values(RISKTYPE) AS RISK_TYPE
values(EVIDENCEDETAILS) AS EVIDENCE_DETAILS

[_JP]

What your subsearch is going to do is create a table of values for POID, IP, IPRISKSCORE, RISKTYPE, EVIDENCEDETAILS and add those to the criteria of your outer search.  It's not adding data to your results stream...rather, just adding criteria (and in your case a bunch of additional AND'ed criteria on your outer search clause).  At first glance the way you structured your subsearch doesn't look like it is doing what you expect it to do since a subsearch in this situation is used to generate dynamic criteria for your search (see more here).

If this overall search is returning the data you want, and just not the fields you already extracted with rex, then you're on the right track...you just need to get those fields generated again.  In that case, do that rex command again before your timechart command.

Also consider if these extractions should be put in a props.conf configuration if you always want them occuring and don't want to use the rex command everywhere.  Don't worry too much - regex extractions like this fail pretty fast and are rather performat if they come across a non-matching event since you're anchoring your match with such a distinct string of "Detected Fraud IP for the User with POID:" versus starting your regex with a greedy term like ".*"

[ITWhisperer]

Is it just POID you want to search by (as suggested by your description) or by POID, IP, IPRISKSCORE, RISKTYPE, EVIDENCEDETAILS (as suggested by your subsearch)?

Also, are POID, IP, IPRISKSCORE, RISKTYPE, EVIDENCEDETAILS already extracted in your main index?

[RobKelley06]

Also POID is not defined as a field in the Main search, if that matters.

[ITWhisperer]

So, how is POID represented in the events you are trying to search?

[RobKelley06 · Accepted solution]

I was able to do a rex and then search after and got what I needed.  Thank you for putting me in the right direction.

index=aws_esf_prod sourcetype="aws:/aws/lambda/lambda-TS4-ESF-transmit-security-activity*" ("journeyId*Q&A" OR "journeyId*OTP Journey" OR "journeyId*fido2_auth" OR "journeyId*mobile_approve") "'policy_version_id'" "device_model" "journeyId"
| rex field=_raw "'poid': '(?<POID>\d+)'"
| search
[
  search index=aws_esf_prod sourcetype=aws:ecs source="*sef/sef-app*" "*Detected Fraud IP*" AND NOT "*56789*" AND NOT "*1234*"
  | rex field=_raw "Detected Fraud IP for the User with POID: (?<POID>\d+) and IP Address : (?<IP>.+) and Risk Score : (?<IPRISKSCORE>\d+) and Risk Type : (?<RISKTYPE>\D+)Evidence Details : (?<EVIDENCEDETAILS>.+)"
  | dedup POID
  | table POID
]

[RobKelley06]

... ', 'poid': '1234567890'}  ...

 The POID from the subsearch would return 1234567890.  But there would be multiple in the table.

[ITWhisperer]

Apart from these being single quotes, this looks like JSON format. If they should be double quotes, perhaps you could extract the field by declaring the sourcetype as being JSON format?

If not, you could try something like this:

index=aws_esf_prod sourcetype="aws:/aws/lambda/lambda-TS4-ESF-transmit-security-activity*" ("journeyId*Q&A" OR "journeyId*OTP Journey" OR "journeyId*fido2_auth" OR "journeyId*mobile_approve") "'policy_version_id'" "device_model" "journeyId"
[
  search index=aws_esf_prod sourcetype=aws:ecs source="*sef/sef-app*" "*Detected Fraud IP*" AND NOT "*56789*" AND NOT "*1234*"
  | rex field=_raw "Detected Fraud IP for the User with POID: (?<POID>\d+) and IP Address : (?<IP>.+) and Risk Score : (?<IPRISKSCORE>\d+) and Risk Type : (?<RISKTYPE>\D+)Evidence Details : (?<EVIDENCEDETAILS>.+)"
  | table POID
  | dedup POID
  | eval search= "'poid': '" + POID + "'"
  | table search
  | format
]

[RobKelley06]

I need to search by POID in the main search, the rest of the fields from the subsearch I can actually remove.  Just realized that I don't need those.

[ITWhisperer]

So, the question remains, is POID already extracted in your main search?

[RobKelley06]

No, it is not. I cannot create a new exatraction for the app as this query will be used in multiple.