"Compress" a multiline search

[atanasmitev]

Hi,

I am trying to compress/optimize a search, spanning multiple lines, see below (obfuscated, but logically the same ). You can see it's hard to process, even in this form:

  index=myindex source="somesource" var1=* | stats count by var2 | search count > 50
| map maxsearches=100 search="search index=myindex source=\"somesource\" var2=$var2$ | stats dc(var1) as thisname by var2| where thisname > 10"
| appendcols
[ index=myindex source="somesource" var1=* | stats count by var2 | search count > 50
  | map maxsearches=100 search="search index=myindex source="somesource" var2=$var2$ | stats count(var1) as thatname by var2" ] 

Logically it looks like this:

PART1| stats dc(var1) as thisname by var2| where thisname > 10 | appendcols [PART1 | stats count(var1) as thatname by var2" ]

And should further be passed to "|table var2, thisname, thatname"

Is there a way, to optimize/compress the search to a form, similar to the one in bold above, perhaps saving PART1 and passing parameters to it ?

[martin_mueller · Accepted solution]

Simplifying that search directly I'd change it to this:

  index=myindex source="somesource" var1=* | stats count by var2 | search count > 50
| map maxsearches=100 search="search index=myindex source=\"somesource\" var2=$var2$ | stats dc(var1) as thisname count(var1) as thatname by var2| where thisname > 10"

However, I think you can change the approach entirely like this:

  index=myindex source=\"somesource\" [ search index=myindex source="somesource" var1=* | stats count by var2 | search count > 50 | fields var2 ]
| stats dc(var1) as thisname count(var1) as thatname by var2 | where thisname > 10

Depending on your data, you may even be able to do this:

index=myindex source=\"somesource\" var1=* | stats count dc(var1) as thisname count(var1) as thatname by var2 | where thisname > 10 AND count > 50 | fields - count

Note, the last approach does change things a tiny bit - before, the last searches didn't require var1=*. Whether this makes sense or not depends on your data and requirements.

[martin_mueller]

You could say you wrote a separate program for printing out each of the letters in "Hello World!" - the original search had a separate search for each of the values in var2, and a separate search for each of the result columns 😛

[atanasmitev]

An unbiased colleague compared my version to a newbie 200-row "Hello world " search.

I tried the last one - WOW !!! Amazing ! It worked like a charm. Thanks a LOT!

Now I need to figure out how you did that, perhaps I should RTFM !!!

Thanks a lot !!!