topic Re: What should I use instead of foreach when iterating? in Splunk Search

What should I use instead of foreach when iterating?

ddrillic — Thu, 12 Apr 2018 17:50:07 GMT

When running the following -

| makeresults 1 
| eval total=0 
| eval server1=host1 
| eval server2=host2
| eval server3=host3

| foreach server*

 [ 
   | forwarderquery server=<<FIELD>> api="/services/server/info" stanza="unix_forwarder"
    contenttype="json" 
| spath path="entry{}" output=entry 
| fields - _raw 
| mvexpand entry 
| spath input=entry 
 ]

I get the error message -
Error in 'foreach' command: Search pipeline may not contain non-streaming commands

What can I do instead?

Re: What should I use instead of foreach when iterating?

elliotproebstel — Thu, 12 Apr 2018 18:17:53 GMT

How about using map instead? Something like this:

| makeresults 
| eval server=host1 
| append 
 [| makeresults
  | eval server=host2 ]
| append
 [| eval server=host3 ] 
 | map maxsearches=3  
  [| forwarderquery server="$server$" api="/services/server/info" stanza="unix_forwarder"
     contenttype="json" 
 | spath path="entry{}" output=entry 
 | fields - _raw 
 | mvexpand entry 
 | spath input=entry ]

Note that you'll need to change the value of maxsearches if you change the number of servers you want to run this over. It's generally not a great idea to use map if you can avoid it, but on a limited scale, it's not terrible. Also, if somebody else has an idea that doesn't use map, then theirs will almost certainly perform better than mine!

Re: What should I use instead of foreach when iterating?

niketn — Thu, 12 Apr 2018 18:22:41 GMT

@ddrillic, the foreach command is supposed to run template version of eval command. So it is probably not the right option for your use case. Instead of having your outer search result as row with several columns i.e. server1, server2... etc, if you can have single column server with several rows host1, host2.. etc, then you can use map command instead to caryy out inner query for each results.
Following is a run anywhere example.

 | makeresults 
 | fields - _time
 | eval server="host1,host2,host3"
 | makemv server delim=","
 | mvexpand server
 | eval total=0
 |  map search="| makeresults
                | fields - _time
                | eval serverList=\"$server$\""

Please confirm if this is what you need. PS: map command is restricted by subsearch limitation and maxsearches=10 is the default option. If your outer search has too many rows, there might be performance issue with the query.

Re: What should I use instead of foreach when iterating?

niketn — Thu, 12 Apr 2018 18:23:32 GMT

@elliotproebstel I am too late with my answer 🙂

Re: What should I use instead of foreach when iterating?

ddrillic — Thu, 12 Apr 2018 18:38:49 GMT

Great but there is a complaint - *Search Factory: Unknown search command 's'. * about server=$server.

Re: What should I use instead of foreach when iterating?

elliotproebstel — Thu, 12 Apr 2018 18:42:23 GMT

Fixed! Try it again.

Re: What should I use instead of foreach when iterating?

ddrillic — Thu, 12 Apr 2018 18:47:25 GMT

Still upset, screaming ; - ) - *Error in 'map' command: Unable to find saved search 'maxsearches=3'. *

Re: What should I use instead of foreach when iterating?

elliotproebstel — Thu, 12 Apr 2018 18:51:24 GMT

Hmmm...It seems like maybe the maxsearches=3 part needs to go at the end? Sorry, I'm air coding at the moment, and clearly not doing that well!

Re: What should I use instead of foreach when iterating?

ddrillic — Thu, 12 Apr 2018 18:53:31 GMT

It's all good ; - ) thank you!

Re: What should I use instead of foreach when iterating?

ddrillic — Thu, 12 Apr 2018 19:03:06 GMT

@niketnilay, that works!

I see -

What's next? ; -)

Re: What should I use instead of foreach when iterating?

elliotproebstel — Thu, 12 Apr 2018 19:08:27 GMT

Put your search inside the search=... section!

Re: What should I use instead of foreach when iterating?

elliotproebstel — Thu, 12 Apr 2018 19:08:52 GMT

Since you only have three values, you won't even really need maxsearches at all. Give it a try without it.

Re: What should I use instead of foreach when iterating?

ddrillic — Thu, 12 Apr 2018 19:17:34 GMT

Perfect -

 | makeresults 
  | fields - _time
  | eval server="host1,host2,host3"
  | makemv server delim=","
  | mvexpand server
  | eval total=0
  |  map search="| forwarderquery server=$server$ api="/services/server/info" stanza="unix_forwarder"
    contenttype="json" 
  | spath path="entry{}" output=entry 
  | fields - _raw 
  | mvexpand entry 
  | spath input=entry
"

It says 3 results but shows only one event ; -)

Re: What should I use instead of foreach when iterating?

elliotproebstel — Thu, 12 Apr 2018 19:19:11 GMT

You'll need to escape any double-quotes inside the mapped search. I suspect it's silently failing because of that. You may also need (escaped) double-quotes around the variable: server=\"$server$\".

Re: What should I use instead of foreach when iterating?

ddrillic — Thu, 12 Apr 2018 19:32:27 GMT

I'll try...

Meanwhile there is another issue -
* [map]: command="forwarderquery", Error : Traceback: 'HTTPSConnectionPool(host='%5C$server%5C$', port=8089): Max retries exceeded with url: /services/server/info *

Re: What should I use instead of foreach when iterating?

niketn — Thu, 12 Apr 2018 19:36:28 GMT

@ddrillic @elliotproebstel, when map command fails it logs the details under Job Inspector with position and error (in most cases ;)). Yes double quote needs to be escaped.

Refer to @woodcock's answer https://answers.splunk.com/answers/543009/field-not-fillled-through-eval-in-map.html where you can use [ ] to use search without having to escape double quotes.

Re: What should I use instead of foreach when iterating?

elliotproebstel — Thu, 12 Apr 2018 19:38:10 GMT

Yeah, that's why I proposed that search structure in my answer - the original query had a lot of double quotes 🙂

Re: What should I use instead of foreach when iterating?

niketn — Thu, 12 Apr 2018 19:50:53 GMT

@ddrillic, I hope you are currently trying query like the following , you also need to escape forward-slashes. Consider map command search string the same way as running a Regular Expression. So, escape all characters with their literal value using backslash i.e. \", \/ etc as needed.

Just FYI %5C is HTML encoded character for Backslash \. So seems like you have missed double quotes in $server$ i.e. \"$server$\"

| makeresults 
| fields - _time 
| eval server="host1,host2,host3" 
| makemv server delim="," 
| mvexpand server 
| eval total=0 
| map search="| forwarderquery server=\"$server$\" api=\"\/services\/server\/info\" stanza=\"unix_forwarder\"
     contenttype=\"json\" 
   | spath path=\"entry{}\" output=entry 
   | fields - _raw 
   | mvexpand entry 
   | spath input=entry"

Further more seems like the error is from the service itself so it is being hit but with wrong value. (I am assuming this is what is happening)

Re: What should I use instead of foreach when iterating?

niketn — Thu, 12 Apr 2018 19:55:49 GMT

😄 For no reasons, I actually prefer using double quotes and escaping all the values using backslash even though query looks much more complicated.

I feel @ddrillic is almost there as the error seems to be from service rather than map command. Hope he makes it through... Its too late for me now.. Anyways he is in good hands 🙂