How to use the map command to match events?

[Ciarán]

Could someone have a look at the following query and see why it does not give me the results I expect based on the documentation of map?

index=portal sourcetype=app:*** source="log" cls=c.b.m.s.SoapServiceClientService Exit event 'ERROR' 
| rex "(?i) .*? \[(?P<ResponseCode>\d+)(?=\])" 
| search ResponseCode=504 
| stats values(ses) as Session 
| map search="search index=portal sourcetype=app:*** source='log' cls='c.b.m.s.SoapClient Webservicecall*' ses=$Session$ | stats first"

So the first search lists all the session ID's for which a certain error occurs.
I want to match this to another event from those sessions which contains the name of the webservice for which the call failed.
The second search finds multiple events in the same session, but with 'stats first' I take the latest, which for now is assumed to be the failing one.
Instead of the events from the second search, I only get events for the first search as results. No table.
So right now my result are events which I would also get when removing the whole map bit and the first stats. Nothing seems to have been added in regards to fields.

The holy grail would allow me to include two more rex commands in the map search to extract two fields. When I add a rex command comparable to the one in the first search, the second search won't run.

[scelikok]

Hi @Ciarán,

Could you please try below search? I think you can achieve your results without subsearch.

index=portal sourcetype=app:*** source="log" cls=c.b.m.s.SoapServiceClientService Exit event 'ERROR' 
| rex "(?i) .*? \[(?P<ResponseCode>\d+)(?=\])" 
| stats first values(ResponseCode) as ResponseCodeList 
| search ResponseCodeList=504 
| rename first(*) as * 
| fields - ResponseCodeList

[Ciarán]

Thanks for the suggestion, but I think you missed a part in my question.
If I am reading this query correctly it only searches in the events which have Exit event 'ERROR' , but the information I need (the webservice name) is not included in these events.
I need to find another event from the same session which occurred a little bit before the error event. That event does contain the name of the web service. 

[richgalloway · Accepted solution]

I think map may not be the solution to this problem.  Have you tried a subsearch?

index=portal sourcetype=app:*** source='log' cls='c.b.m.s.SoapClient Webservicecall*' [
index=portal sourcetype=app:*** source="log" cls=c.b.m.s.SoapServiceClientService Exit event 'ERROR' 
| rex "(?i) .*? \[(?P<ResponseCode>\d+)(?=\])" 
| search ResponseCode=504 
| fields ses
| format ]

[Ciarán]

Thanks for your advice.
Unfortunately I did not get any results with this query where I would expect 289 occurences of this in the time frame.
I therefor edited your query following what is done in Example 2 of Use a subsearch - Splunk Documentation but I still get no results.
The subsearch does return 289 results as expected.

index=portal sourcetype=app:*** source='log' cls='c.b.m.s.SoapClient Webservicecall*' [
index=portal sourcetype=app:*** source="log" cls=c.b.m.s.SoapServiceClientService Exit event 'ERROR' 
| rex "(?i) .*? \[(?P<ResponseCode>\d+)(?=\])" 
| search ResponseCode=504 
| table ses ]

[ITWhisperer]

I am assuming you corrected the syntax?

index=portal sourcetype=app:*** source='log' cls='c.b.m.s.SoapClient Webservicecall*' [ search
index=portal sourcetype=app:*** source="log" cls=c.b.m.s.SoapServiceClientService Exit event 'ERROR' 
| rex "(?i) .*? \[(?P<ResponseCode>\d+)(?=\])" 
| search ResponseCode=504 
| table ses ]

Using one of the values for ses returned by

index=portal sourcetype=app:*** source="log" cls=c.b.m.s.SoapServiceClientService Exit event 'ERROR' 
| rex "(?i) .*? \[(?P<ResponseCode>\d+)(?=\])" 
| search ResponseCode=504 
| table ses

does 

index=portal sourcetype=app:*** source='log' cls='c.b.m.s.SoapClient Webservicecall*' ses="value returned"

return any results?

[Ciarán]

I did fix the syntax 🙂

Indeed it does not return any results and I have concluded that this might be because some application have not upgraded to the new logging standards so some expected logging is missing.
Testing it on a different timeframe in which I am sure there is a updated logging did work!

Thanks!

[Ciarán]

Oh no! I was too quick to reply.
So the loose queries work. Returning 7 events for any one session.
But the subsearch version does not work.

In the end I would like to not have 7 events per session but the info from 1 specific event (the latest one from the x events).

[ITWhisperer]

index=portal sourcetype=app:*** source='log' cls='c.b.m.s.SoapClient Webservicecall*' [ search
index=portal sourcetype=app:*** source="log" cls=c.b.m.s.SoapServiceClientService Exit event 'ERROR' 
| rex "(?i) .*? \[(?P<ResponseCode>\d+)(?=\])" 
| search ResponseCode=504 
| dedup ses
| table ses ]
| sort 0 - _time
| dedup ses

[Ciarán]

Thanks for thinking with me.
Could you give a bit more explanation with your query examples?

Unfortunately this did not solve the problem.

I have a subsearch result which does not contain duplicates. (it sometimes does, but right now it does not)

The composed query with subsearch says it has zero results, so the sort and dedup seem useless?

But to clarify my wishes. I need the latest event because the other results are not relevant to this search but I have no way to match them with the search other than they occur within a certain time frame before the error event. Will this sort and dedup give me the latest? Or does dedup result in a random event?

[ITWhisperer]

The subsearch essentially filters the base search by extending it with (( ses="xyz") OR (ses="abc"))

The dedup in the subsearch stops you getting ((ses="xyz") OR (ses="xyz") OR (ses="abc"))

The sort 0 - _time puts the result from the filtered base search in reverse chronological order

The dedup takes the first occurrence of each ses in the results (i.e. the latest by ses)

If there are no results, there is nothing to sort or dedup.

[Ciarán]

So the subsearch has 14 results.
The first hit of the subsearch has 7 results on the main search.
But when I run the composed search with the subsearch in it I get 0 results.

What's going on?

[ITWhisperer]

How many events are scanned (not returned) by the subsearch?

Subsearches are limited to 50,000 events - if you have more than 50,000 events returned by the initial search of the subsearch, your results are probably non-deterministic.

In this instance, you may need to break up your subsearch into multiple chunks.

You can test the theory by reducing the timescale of the search to see if that resolves the issue.

[Ciarán]

Thanks for another helping hand!

So apparently the amount of events was not the issue. If I'm not mistaken I saw the number 38.000 something somewhere in the job inspect.
But the job inspection did tell me that it had trouble with the wild card in my main search. So I replaced that with a string match instead of a field match so I could circumvent the wild card and now it works!

Thanks so much!

[ITWhisperer]

It is often good to look at what the job inspector is telling you, it is not just for show 😁