Trouble looping through table and performing a subsearch for each item.

[Ara]

I am trying to loop over a table and perform a subsearch for each item. I can confirm I am generating the first table with correct values. However the subsearch portion is not returning any results. 

Can someone help me figure out where my query is wrong? Would be very much appreciated!

index=xyz "someString"
| rex field=msg "DEBUG\s+\|\s+(?<traceid>[a-f0-9-]{36})"
| table traceid
| map search="search index=xyz \"$traceid$\" AND \"REQUEST BODY\"
| rex field=msg \"artifact_guid\":\"(?<artifact_guid>[a-f0-9-]{36})\"
| rex field=msg \"email_address\":\"(?<email_address>[^\"]+)\"
| table traceid, artifact_guid, email_address"

[livehybrid]

Hi @Ara 

As others have said, I dont think a map is well placed here, you might find the following useful, which determines if your "someString" is present and filters those which have this.

index=xyz 
| rex field=msg "DEBUG\s+\|\s+(?<traceid>[a-f0-9-]{36})"
| rex field=msg \"artifact_guid\":\"(?<artifact_guid>[a-f0-9-]{36})\"
| rex field=msg \"email_address\":\"(?<email_address>[^\"]+)\"
| eval isInteresting=IF(searchmatch("someString"),1,0) | stats max(isInteresting) as isInteresting, values(artifact_guid) as artifact_guid, values(email_address) as email_address by traceid
| where isInteresting>0

You might be able to simplify by dealing with the rex separately with field extractions but the premise here is that you use searchmatch to check for your interesting string, then filter out events which do not have it after matching the traceId with the other fields you want.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[PickleRick]

While I fully agree with @yuanliu and @bowesmana that map is most probably not the way to go in this case there is an obvious lack of/improper use of quotes.

If you unescape your mapped search you'll get

search index=xyz "$traceid$" AND "REQUEST BODY"
| rex field=msg "artifact_guid":"(?<artifact_guid>[a-f0-9-]{36})"
| rex field=msg "email_address":"(?<email_address>[^"]+)"
| table traceid, artifact_guid, email_address

This is not the right syntax for rex command.

As far as I remember map won't pass the errors from the mapped search to your outer search - it will just not yield any results. Hence your behaviour.

[bowesmana · Accepted solution]

map may not be the best solution here - as to why it's not working, if you just have the map search and nothing after does it return results?

Why do you have escaped quotes round the \"$traceid$\" - is your trace id string quoted in the data?

Note that unless traceid exists in your map search results, your table statement will have no value for traceid.

However, given the somewhat complex quote escaping you are doing to parse data from what appears to be JSON data, I expect the problem is somewhere there. You'd need to give a data example.

However, map is generally not a great way to solve a problem, because it's default limit is 10 results and if you do more, each search runs as a separate sequential search, so adds additional load on the indexes/search head.

You may be better off using a simple stats command to join the data, e.g.

index=xyz "someString" OR "REQUEST BODY"
``` Extract the traceid fron the DBUG message containing someString and then from the request body ```
| rex field=msg "DEBUG\s+\|\s+(?<debug_traceid>[a-f0-9-]{36})"
| rex field=field_containing_traceid_in_request_body "(?<body_traceid>[a-f0-9-]{36})"
| eval traceid=coalesce(debug_traceid, body_traceid)
| rex field=msg "artifact_guid\":\"(?<artifact_guid>[a-f0-9-]{36})"
| rex field=msg "email_address\":\"(?<email_address>[^\"]+)"
| stats values(debug_traceid) as debug_traceid values(artifact_guid) as artifact_guid values(email_address) as email_address by traceid
| where isnotnull(debug_traceid)

So this searches both data sets, gets the traceid from both event types and then collects the artifact and email from the body and joins the data sets and then filters out those that do not have a matching someString event.

[Ara]

Thank you for your response! This solution worked. I was not aware of the 10-result limit with map.

[bowesmana]

Note: The 10 result limit for map is a soft limit, you can go more with the maxsearches=X setting, but again, map is rarely useful.

Glad you got a working solution.

[yuanliu]

Totally agree with @bowesmana that there is no benefit of using map in this use case.  I also have a suggestion: Do not bother with rex.  Looking at your regular expression, it is all but certain that the field msg contains JSON.  This would be much simpler:

index=xyz ("someString" OR "REQUEST BODY")
``` Extract the traceid fron the DBUG message containing someString and then from the request body ```
| rex field=msg "DEBUG\s+\|\s+(?<debug_traceid>[a-f0-9-]{36})"
| rex field=field_containing_traceid_in_request_body "(?<body_traceid>[a-f0-9-]{36})"
| eval traceid=coalesce(debug_traceid, body_traceid)
| spath input=msg
| stats values(debug_traceid) as debug_traceid values(artifact_guid) as artifact_guid values(email_address) as email_address by traceid
| where isnotnull(debug_traceid)