Splunk Search

How to use the map command to match events?

Ciarán
Explorer

Could someone have a look at the following query and see why it does not give me the results I expect based on the documentation of map?

 

index=portal sourcetype=app:*** source="log" cls=c.b.m.s.SoapServiceClientService Exit event 'ERROR' 
| rex "(?i) .*? \[(?P<ResponseCode>\d+)(?=\])" 
| search ResponseCode=504 
| stats values(ses) as Session 
| map search="search index=portal sourcetype=app:*** source='log' cls='c.b.m.s.SoapClient Webservicecall*' ses=$Session$ | stats first"

 


So the first search lists all the session ID's for which a certain error occurs.
I want to match this to another event from those sessions which contains the name of the webservice for which the call failed.
The second search finds multiple events in the same session, but with 'stats first' I take the latest, which for now is assumed to be the failing one.
Instead of the events from the second search, I only get events for the first search as results. No table.
So right now my result are events which I would also get when removing the whole map bit and the first stats. Nothing seems to have been added in regards to fields.

The holy grail would allow me to include two more rex commands in the map search to extract two fields. When I add a rex command comparable to the one in the first search, the second search won't run.

Labels (3)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

I think map may not be the solution to this problem.  Have you tried a subsearch?

index=portal sourcetype=app:*** source='log' cls='c.b.m.s.SoapClient Webservicecall*' [
index=portal sourcetype=app:*** source="log" cls=c.b.m.s.SoapServiceClientService Exit event 'ERROR' 
| rex "(?i) .*? \[(?P<ResponseCode>\d+)(?=\])" 
| search ResponseCode=504 
| fields ses
| format ]
---
If this reply helps you, Karma would be appreciated.

View solution in original post

scelikok
SplunkTrust
SplunkTrust

Hi @Ciarán,

Could you please try below search? I think you can achieve your results without subsearch.

index=portal sourcetype=app:*** source="log" cls=c.b.m.s.SoapServiceClientService Exit event 'ERROR' 
| rex "(?i) .*? \[(?P<ResponseCode>\d+)(?=\])" 
| stats first values(ResponseCode) as ResponseCodeList 
| search ResponseCodeList=504 
| rename first(*) as * 
| fields - ResponseCodeList

 

If this reply helps you an upvote and "Accept as Solution" is appreciated.
0 Karma

Ciarán
Explorer

Thanks for the suggestion, but I think you missed a part in my question.
If I am reading this query correctly it only searches in the events which have Exit event 'ERROR' , but the information I need (the webservice name) is not included in these events.
I need to find another event from the same session which occurred a little bit before the error event. That event does contain the name of the web service. 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

I think map may not be the solution to this problem.  Have you tried a subsearch?

index=portal sourcetype=app:*** source='log' cls='c.b.m.s.SoapClient Webservicecall*' [
index=portal sourcetype=app:*** source="log" cls=c.b.m.s.SoapServiceClientService Exit event 'ERROR' 
| rex "(?i) .*? \[(?P<ResponseCode>\d+)(?=\])" 
| search ResponseCode=504 
| fields ses
| format ]
---
If this reply helps you, Karma would be appreciated.

Ciarán
Explorer

Thanks for your advice.
Unfortunately I did not get any results with this query where I would expect 289 occurences of this in the time frame.
I therefor edited your query following what is done in Example 2 of Use a subsearch - Splunk Documentation but I still get no results.
The subsearch does return 289 results as expected.

index=portal sourcetype=app:*** source='log' cls='c.b.m.s.SoapClient Webservicecall*' [
index=portal sourcetype=app:*** source="log" cls=c.b.m.s.SoapServiceClientService Exit event 'ERROR' 
| rex "(?i) .*? \[(?P<ResponseCode>\d+)(?=\])" 
| search ResponseCode=504 
| table ses ]
0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

I am assuming you corrected the syntax?

index=portal sourcetype=app:*** source='log' cls='c.b.m.s.SoapClient Webservicecall*' [ search
index=portal sourcetype=app:*** source="log" cls=c.b.m.s.SoapServiceClientService Exit event 'ERROR' 
| rex "(?i) .*? \[(?P<ResponseCode>\d+)(?=\])" 
| search ResponseCode=504 
| table ses ]

Using one of the values for ses returned by

index=portal sourcetype=app:*** source="log" cls=c.b.m.s.SoapServiceClientService Exit event 'ERROR' 
| rex "(?i) .*? \[(?P<ResponseCode>\d+)(?=\])" 
| search ResponseCode=504 
| table ses

does 

index=portal sourcetype=app:*** source='log' cls='c.b.m.s.SoapClient Webservicecall*' ses="value returned"

return any results?

Ciarán
Explorer

I did fix the syntax 🙂

Indeed it does not return any results and I have concluded that this might be because some application have not upgraded to the new logging standards so some expected logging is missing.
Testing it on a different timeframe in which I am sure there is a updated logging did work!

Thanks!

0 Karma

Ciarán
Explorer

Oh no! I was too quick to reply.
So the loose queries work. Returning 7 events for any one session.
But the subsearch version does not work.

In the end I would like to not have 7 events per session but the info from 1 specific event (the latest one from the x events).

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust
index=portal sourcetype=app:*** source='log' cls='c.b.m.s.SoapClient Webservicecall*' [ search
index=portal sourcetype=app:*** source="log" cls=c.b.m.s.SoapServiceClientService Exit event 'ERROR' 
| rex "(?i) .*? \[(?P<ResponseCode>\d+)(?=\])" 
| search ResponseCode=504 
| dedup ses
| table ses ]
| sort 0 - _time
| dedup ses
0 Karma

Ciarán
Explorer

Thanks for thinking with me.
Could you give a bit more explanation with your query examples?

Unfortunately this did not solve the problem.

I have a subsearch result which does not contain duplicates. (it sometimes does, but right now it does not)

The composed query with subsearch says it has zero results, so the sort and dedup seem useless?

But to clarify my wishes. I need the latest event because the other results are not relevant to this search but I have no way to match them with the search other than they occur within a certain time frame before the error event. Will this sort and dedup give me the latest? Or does dedup result in a random event?

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

The subsearch essentially filters the base search by extending it with (( ses="xyz") OR (ses="abc"))

The dedup in the subsearch stops you getting ((ses="xyz") OR (ses="xyz") OR (ses="abc"))

The sort 0 - _time puts the result from the filtered base search in reverse chronological order

The dedup takes the first occurrence of each ses in the results (i.e. the latest by ses)

If there are no results, there is nothing to sort or dedup.

0 Karma

Ciarán
Explorer

So the subsearch has 14 results.
The first hit of the subsearch has 7 results on the main search.
But when I run the composed search with the subsearch in it I get 0 results.

What's going on?

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

How many events are scanned (not returned) by the subsearch?

Subsearches are limited to 50,000 events - if you have more than 50,000 events returned by the initial search of the subsearch, your results are probably non-deterministic.

In this instance, you may need to break up your subsearch into multiple chunks.

You can test the theory by reducing the timescale of the search to see if that resolves the issue.

Ciarán
Explorer

Thanks for another helping hand!

So apparently the amount of events was not the issue. If I'm not mistaken I saw the number 38.000 something somewhere in the job inspect.
But the job inspection did tell me that it had trouble with the wild card in my main search. So I replaced that with a string match instead of a field match so I could circumvent the wild card and now it works!

Thanks so much!

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

It is often good to look at what the job inspector is telling you, it is not just for show 😁

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...