Foreach in Multisearch

[IRHM73]

Hi, I wonder whether someone can help me please.

I've put together the query below using the foreach command, which, although I've read a lot of posts, I've not really used, or if truth be known understood a great deal.

| multisearch
[ search `gateway_wmf(ClientRequest)` path=*vat*]
[ search `wso2_wmf(RequestCompleted)` "request.detail.apiContext"=*test]
| eval RequestID=coalesce('request.tags.X-Request-ID','requestID')
| dedup eventId
| rename request.detail.applicationProductionClientId as ClientID response.detail.statusCode AS statusCode
| foreach clientHeaders.test* [eval header='<<FIELD>>']
| stats count(header) by RequestID 

The query runs, but there is no new field called "header" created and hence I don't receive my stats count at the end of the query.

Could someone perhaps have a look a this please and offer some guidance on where I've gone wrong and a brief explanation of the 'foreach' command.

Many thanks and kind regards

Chris

[tiagofbmm]

If there is no header field on the search result I guess there is no fields clientHeaders.test with a suffix.

Check this dummy example, where only one field called header is created, which seems to not have much logic when framing this with the foreach command

| makeresults | eval clientHeaders.test1=1, clientHeaders.test2=2 | foreach clientHeaders.test* [eval header='<<FIELD>>']

If you want a field header_ for each of the clientHeaders.test, you can use

| makeresults | eval clientHeaders.test1=1, clientHeaders.test2=2 | foreach clientHeaders.test* [eval header_<<MATCHSTR>>=<<FIELD>>']

In my opinion it doesn't seem to be what you want to achieve from what I get from your query. Would you put here an anonymyzed example of existing fields at this point:

| multisearch
[ search gateway_wmf(ClientRequest) path=vat]
[ search wso2_wmf(RequestCompleted) "request.detail.apiContext"=*test]
| eval RequestID=coalesce('request.tags.X-Request-ID','requestID')
| dedup eventId
| rename request.detail.applicationProductionClientId as ClientID response.detail.statusCode AS statusCode

And get us what you want to get in the end?

[IRHM73]

Hi @tiagofbmm . Thank you for coming back to me with this.

So some examples of the fields are:

clientHeaders.test-client-colour-depth
clientHeaders.test-client-device-id
clientHeaders.test-client-device-ip
clientHeaders.test-test-scenario

But this list is not exhaustive. It will increase but I would like to 'futureproof' the query so I don't need to constantly update when new fields appear with this naming convention, if possible please.

And for the columns, I'd like if possible to be as follows:

RequestID Time Multiple columns where fields start with clientHeaders.test
1234 15/05 Count for each column heading

I hope this makes sense and once again many thanks.

Kind Regards

Chris

[richgalloway · Accepted solution]

The foreach command repeats the commands in its subsearch for each field (not value) that matches the first argument. In your query, the eval command will be invoked for all of the fields with names beginning with 'clientHeaders.test'. If there are no such fields then the eval is not executed at all.

[IRHM73]

Hi @richgalloway, thank you for coming back to me with this.

I've managed to put a solution together as follows:

| multisearch
[ search `gateway_wmf(ClientRequest)` path=*test*]
[ search `wso2_wmf(RequestCompleted)` "request.detail.apiContext"=*test]
| eval RequestID=coalesce('request.tags.X-Request-ID','requestID')
| eval header=""
| foreach clientHeaders.test* [eval header='<<FIELD>>']
| bucket span=10s _time
| stats count(header) by RequestID _time header

The problem I have, and this is through lack of knowledge, I didn't realise that it extracted the field value rather than the fieldname.

Could you tell me please is there a way to count the field headers rather than the values?

Many thanks and kind regards

Chris

[richgalloway]

It's not very elegant, but try this.

| multisearch
[ search `gateway_wmf(ClientRequest)` path=*test*]
[ search `wso2_wmf(RequestCompleted)` "request.detail.apiContext"=*test]
| eval RequestID=coalesce('request.tags.X-Request-ID','requestID')
| eval header=""
| foreach clientHeaders.test* [eval header=header+test<<MATCHSTR>>]
| bucket span=10s _time
| stats count(header) by RequestID _time header

[IRHM73]

Hi @richgalloway.

Thank you for coming back to me with this. May I just check, in this

[eval header=header+test<<MATCHSTR>>].

The test element. Where is this taken from?

Many thanks and kind regards

Chris

[richgalloway]

It comes from the query in your question. You had eval header=<<FIELD>> where FIELD is clientHeaders.test* so I used MATCHSTR to get the * part and inserted 'test' manually.

[IRHM73]

Hi @richgalloway. Thank you. I thought that was the case.

Unfortunately it only gives me two columns, 1 called header and the the other count(header), in addition to the time and RequestID.

Many thanks and regards

Chris

[richgalloway]

That is more than your original query produced. What exactly do you want in the results?

[IRHM73]

Hi.

Running them side by side, the results are table layout are the same from what I can see.

Here is the layout I'd like if possible please:

RequestID Time Multiple columns where fields start with clientHeaders.test
1234 15/05 Count for each column heading

These are examples of the fieldnames:
clientHeaders.test-client-device-id
clientHeaders.test-client-device-ip
clientHeaders.test-test-scenario

But this list is not exhaustive. It will increase but I would like to 'futureproof' the query so I don't need to constantly update when new fields appear with this naming convention, if possible please.

Many thanks and regards

Chris

[richgalloway]

Try this.

| multisearch
[ search `gateway_wmf(ClientRequest)` path=*test*]
[ search `wso2_wmf(RequestCompleted)` "request.detail.apiContext"=*test]
| eval RequestID=coalesce('request.tags.X-Request-ID','requestID')
| eval header=""
| foreach clientHeaders.test* [eval header=header+test<<MATCHSTR>>]
| table RequestID _time header

[IRHM73]

Hi @richgalloway. Thank you for posting this.

As you would expect the table has the 3 columns, but unfortunately the "header" column is blank.

Many thanks and kind regards

Chris

[richgalloway]

What do you get when you change the table command to | table RequestID _time header clientHeaders.test*?

[IRHM73]

Hi @richgalloway . That makes a huge difference.

I now have the columns I want.....it's great.

I just now need to work out had to incorporate the counts but I think I can work with that.

Thank you so much for all your time and trouble, it's sincerely appreciated.

Kind Regards

Chris