Splunk Search

Foreach in Multisearch

IRHM73
Motivator

Hi, I wonder whether someone can help me please.

I've put together the query below using the foreach command, which, although I've read a lot of posts, I've not really used, or if truth be known understood a great deal.

| multisearch
[ search `gateway_wmf(ClientRequest)` path=*vat*]
[ search `wso2_wmf(RequestCompleted)` "request.detail.apiContext"=*test]
| eval RequestID=coalesce('request.tags.X-Request-ID','requestID')
| dedup eventId
| rename request.detail.applicationProductionClientId as ClientID response.detail.statusCode AS statusCode
| foreach clientHeaders.test* [eval header='<<FIELD>>']
| stats count(header) by RequestID 

The query runs, but there is no new field called "header" created and hence I don't receive my stats count at the end of the query.

Could someone perhaps have a look a this please and offer some guidance on where I've gone wrong and a brief explanation of the 'foreach' command.

Many thanks and kind regards

Chris

0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

The foreach command repeats the commands in its subsearch for each field (not value) that matches the first argument. In your query, the eval command will be invoked for all of the fields with names beginning with 'clientHeaders.test'. If there are no such fields then the eval is not executed at all.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

0 Karma

tiagofbmm
Influencer

If there is no header field on the search result I guess there is no fields clientHeaders.test with a suffix.

Check this dummy example, where only one field called header is created, which seems to not have much logic when framing this with the foreach command

| makeresults | eval clientHeaders.test1=1, clientHeaders.test2=2 | foreach clientHeaders.test* [eval header='<<FIELD>>']

If you want a field header_ for each of the clientHeaders.test, you can use

| makeresults | eval clientHeaders.test1=1, clientHeaders.test2=2 | foreach clientHeaders.test* [eval header_<<MATCHSTR>>=<<FIELD>>']

In my opinion it doesn't seem to be what you want to achieve from what I get from your query. Would you put here an anonymyzed example of existing fields at this point:

| multisearch
[ search gateway_wmf(ClientRequest) path=vat]
[ search wso2_wmf(RequestCompleted) "request.detail.apiContext"=*test]
| eval RequestID=coalesce('request.tags.X-Request-ID','requestID')
| dedup eventId
| rename request.detail.applicationProductionClientId as ClientID response.detail.statusCode AS statusCode

And get us what you want to get in the end?

0 Karma

IRHM73
Motivator

Hi @tiagofbmm . Thank you for coming back to me with this.

So some examples of the fields are:

clientHeaders.test-client-colour-depth
clientHeaders.test-client-device-id
clientHeaders.test-client-device-ip
clientHeaders.test-test-scenario

But this list is not exhaustive. It will increase but I would like to 'futureproof' the query so I don't need to constantly update when new fields appear with this naming convention, if possible please.

And for the columns, I'd like if possible to be as follows:

RequestID Time Multiple columns where fields start with clientHeaders.test
1234 15/05 Count for each column heading

I hope this makes sense and once again many thanks.

Kind Regards

Chris

0 Karma

richgalloway
SplunkTrust
SplunkTrust

The foreach command repeats the commands in its subsearch for each field (not value) that matches the first argument. In your query, the eval command will be invoked for all of the fields with names beginning with 'clientHeaders.test'. If there are no such fields then the eval is not executed at all.

---
If this reply helps you, Karma would be appreciated.
0 Karma

IRHM73
Motivator

Hi @richgalloway, thank you for coming back to me with this.

I've managed to put a solution together as follows:

| multisearch
[ search `gateway_wmf(ClientRequest)` path=*test*]
[ search `wso2_wmf(RequestCompleted)` "request.detail.apiContext"=*test]
| eval RequestID=coalesce('request.tags.X-Request-ID','requestID')
| eval header=""
| foreach clientHeaders.test* [eval header='<<FIELD>>']
| bucket span=10s _time
| stats count(header) by RequestID _time header

The problem I have, and this is through lack of knowledge, I didn't realise that it extracted the field value rather than the fieldname.

Could you tell me please is there a way to count the field headers rather than the values?

Many thanks and kind regards

Chris

0 Karma

richgalloway
SplunkTrust
SplunkTrust

It's not very elegant, but try this.

| multisearch
[ search `gateway_wmf(ClientRequest)` path=*test*]
[ search `wso2_wmf(RequestCompleted)` "request.detail.apiContext"=*test]
| eval RequestID=coalesce('request.tags.X-Request-ID','requestID')
| eval header=""
| foreach clientHeaders.test* [eval header=header+test<<MATCHSTR>>]
| bucket span=10s _time
| stats count(header) by RequestID _time header
---
If this reply helps you, Karma would be appreciated.
0 Karma

IRHM73
Motivator

Hi @richgalloway.

Thank you for coming back to me with this. May I just check, in this

[eval header=header+test<<MATCHSTR>>].

The test element. Where is this taken from?

Many thanks and kind regards

Chris

0 Karma

richgalloway
SplunkTrust
SplunkTrust

It comes from the query in your question. You had eval header=<<FIELD>> where FIELD is clientHeaders.test* so I used MATCHSTR to get the * part and inserted 'test' manually.

---
If this reply helps you, Karma would be appreciated.
0 Karma

IRHM73
Motivator

Hi @richgalloway. Thank you. I thought that was the case.

Unfortunately it only gives me two columns, 1 called header and the the other count(header), in addition to the time and RequestID.

Many thanks and regards

Chris

0 Karma

richgalloway
SplunkTrust
SplunkTrust

That is more than your original query produced. What exactly do you want in the results?

---
If this reply helps you, Karma would be appreciated.
0 Karma

IRHM73
Motivator

Hi.

Running them side by side, the results are table layout are the same from what I can see.

Here is the layout I'd like if possible please:

RequestID Time Multiple columns where fields start with clientHeaders.test
1234 15/05 Count for each column heading

These are examples of the fieldnames:
clientHeaders.test-client-device-id
clientHeaders.test-client-device-ip
clientHeaders.test-test-scenario

But this list is not exhaustive. It will increase but I would like to 'futureproof' the query so I don't need to constantly update when new fields appear with this naming convention, if possible please.

Many thanks and regards

Chris

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Try this.

| multisearch
[ search `gateway_wmf(ClientRequest)` path=*test*]
[ search `wso2_wmf(RequestCompleted)` "request.detail.apiContext"=*test]
| eval RequestID=coalesce('request.tags.X-Request-ID','requestID')
| eval header=""
| foreach clientHeaders.test* [eval header=header+test<<MATCHSTR>>]
| table RequestID _time header
---
If this reply helps you, Karma would be appreciated.
0 Karma

IRHM73
Motivator

Hi @richgalloway. Thank you for posting this.

As you would expect the table has the 3 columns, but unfortunately the "header" column is blank.

Many thanks and kind regards

Chris

0 Karma

richgalloway
SplunkTrust
SplunkTrust

What do you get when you change the table command to | table RequestID _time header clientHeaders.test*?

---
If this reply helps you, Karma would be appreciated.
0 Karma

IRHM73
Motivator

Hi @richgalloway . That makes a huge difference.

I now have the columns I want.....it's great.

I just now need to work out had to incorporate the counts but I think I can work with that.

Thank you so much for all your time and trouble, it's sincerely appreciated.

Kind Regards

Chris

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...