Token used in search is verbose rather than the evaluated value?

[yaharga]

Trying to use a token in a rex, but can't. I'm setting the token as follows (token_keywords_mv is a mv):

<set token="token_rex">mvjoin(mvmap('token_keywords_mv',"(?&gt;".'token_keywords_mv'."&lt;".'token_keywords_mv'."+?)"), "|")</set>

When I use it in a rex command, it turns verbose:

... | rex field=_raw '(?i)$token_rex$' 

It gives me the following error:

 Error in 'rex' command: Encountered the following error while compiling the regex ''(?i)mvjoin(mvmap('token_keywords_mv'': Regex: missing closing parenthesis.  

 When I set the token as the results of the eval, as in the following, it works:

<set token="token_rex">(?&lt;lorem&gt;lorem+?)|(?&lt;ipsum&gt;ipsum+?)|(?&lt;situs&gt;situs+?)</set>

[ITWhisperer]

Try using eval rather than set when you want a function executed

<eval token="token_rex">mvjoin(mvmap('token_keywords_mv',"(?&gt;".'token_keywords_mv'."&lt;".'token_keywords_mv'."+?)"), "|")</eval>

Having said that, not all functions work and I have a feeling the mv functions may fall foul of this.

[yaharga]

I tried eval and it also doesn't work. If mv functions do not work, is there an alternative or a hack? Can a subsearch be leveraged?

Perhaps there's a way to force the eval in front of the rex? I know of eval().

What about ignoring the token and hardcoding the code in front of the rex?

... | rex field=_raw '(?i) mvjoin(mvmap('token_keywords_mv',"(?>".'token_keywords_mv'."<".'token_keywords_mv'."+?)"), "|")'

[ITWhisperer]

It looks like mvmap is the culprit in this instance - try it with mvzip

<eval token="token_rex">"(&lt;".mvjoin(mvzip('token_keywords_mv','token_keywords_mv',"&gt;"),"+?)|(?&lt;")."+?)"</eval>

[yaharga]

mvzip gets converted into an mvmap in the end.

Error in 'rex' command: Encountered the following error while compiling the regex 'mvjoin(mvmap('token_keywords_mv'': Regex: missing closing parenthesis.

---

[ITWhisperer]

Can you share your SimpleXML?

[yaharga]

<dashboard theme="dark">
  <init>
    <set token="token_keywords_mv">split("lorem,ipsum,situs", ",")</set>
    <set token="token_keywords_starred">"*".mvjoin($token_keywords_mv$, "* OR *")."*"</set>
    <set token="token_rex">"(?i)".mvjoin(mvmap('token_keywords_mv', "(?&lt;".'token_keywords_mv'."&gt;".'token_keywords_mv'."+?)"), "|")</set>
    <set token="token_raw">(?&lt;lorem&gt;lorem+?)|(?&lt;ipsum&gt;ipsum+?)|(?&lt;situs&gt;situs+?)</set>
  </init>
  <label>Test Search</label>
  <description>Multivalue</description>
  <row>
    <panel>
      <table>
        <search>
          <query>index=* 
    [| makeresults 
    | eval string_search=$token_keywords_starred$ 
    | return $string_search] 
| rex field=_raw $token_raw|s$ 
| stats count(lorem) AS Lorem, count(ipsum) AS Ipsum, count(situs) AS Situs
| eval header="Count" 
| transpose column_name="String" header_field=header</query>
          <earliest>-24h@h</earliest>
          <latest>now</latest>
        </search>
        <option name="drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </table>
    </panel>
  </row>
  <row>
    <panel>
      <table>
        <title>ECHO</title>
        <search>
          <query>| localop 
| makeresults 
| eval token_keywords_starred=$token_keywords_starred$, token_keywords_mv=$token_keywords_mv$, token_rex=$token_rex$, token_raw=$token_raw|s$</query>
          <earliest>-24h@h</earliest>
          <latest>now</latest>
        </search>
        <option name="drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </table>
    </panel>
  </row>
</dashboard>

[ITWhisperer]

You are still using set and you are still using mvmap - why not try using eval and mvzip as I suggested?

[yaharga]

I already did the eval, but thought that it produced "mvmap" in the error message, so it's just doing it through another method. I've tried it, but the outcome is the same:

<dashboard theme="dark">
  <init>
    <set token="token_keywords_mv">split("lorem,ipsum,situs", ",")</set>
    <set token="token_keywords_starred">"*".mvjoin($token_keywords_mv$, "* OR *")."*"</set>
    <set token="token_rex">"(?i)".mvjoin(mvmap('token_keywords_mv', "(?&lt;".'token_keywords_mv'."&gt;".'token_keywords_mv'."+?)"), "|")</set>
    <eval token="token_rex_eval">"(&lt;".mvjoin(mvzip('token_keywords_mv','token_keywords_mv',"&gt;"),"+?)|(?&lt;")."+?)"</eval>
    <set token="token_raw">(?&lt;lorem&gt;lorem+?)|(?&lt;ipsum&gt;ipsum+?)|(?&lt;situs&gt;situs+?)</set>
  </init>
  <label>Test Search</label>
  <description>Multivalue</description>
  <row>
    <panel>
      <table>
        <search>
          <query>index=* 
    [| makeresults 
    | eval string_search=$token_keywords_starred$ 
    | return $string_search] 
| rex field=_raw $token_raw|s$ 
| stats count(lorem) AS Lorem, count(ipsum) AS Ipsum, count(situs) AS Situs
| eval header="Count" 
| transpose column_name="String" header_field=header</query>
          <earliest>-24h@h</earliest>
          <latest>now</latest>
        </search>
        <option name="drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </table>
    </panel>
  </row>
  <row>
    <panel>
      <table>
        <title>ECHO</title>
        <search>
          <query>| localop 
| makeresults 
| eval token_keywords_starred=$token_keywords_starred$, token_keywords_mv=$token_keywords_mv$, token_rex=$token_rex$, token_raw=$token_raw|s$, token_rex_eval=$token_rex_eval|s$</query>
          <earliest>-24h@h</earliest>
          <latest>now</latest>
        </search>
        <option name="drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </table>
    </panel>
  </row>
</dashboard>

If I replace $token_raw|s$ with any of the other tokens, it stops working.

[ITWhisperer · Accepted solution]

Anywhere you have code to be executed to define a token you need to use eval - if you are just defining it with a string, you can use set

    <eval token="token_keywords_mv">split("lorem,ipsum,situs", ",")</eval>
    <eval token="token_keywords_starred">"*".mvjoin($token_keywords_mv$, "* OR *")."*"</eval>
    <eval token="token_rex">"(?i)".mvjoin(mvmap('token_keywords_mv', "(?&lt;".'token_keywords_mv'."&gt;".'token_keywords_mv'."+?)"), "|")</eval>
    <eval token="token_rex_eval">"(&lt;".mvjoin(mvzip('token_keywords_mv','token_keywords_mv',"&gt;"),"+?)|(?&lt;")."+?)"</eval>
    <set token="token_raw">(?&lt;lorem&gt;lorem+?)|(?&lt;ipsum&gt;ipsum+?)|(?&lt;situs&gt;situs+?)</set>

You will see that mvmap doesn't work but mvzip does - having said that, it does depend on which version of splunk you are using.

[yaharga]

I knew eval was meant for evaluation and set was not the right thing to use when evaluating, but I went for using set because of two reasons:

• In some situations I did not have to resort to double quoting my tokens.
• For some reason on some dashboards using eval didn't set the token, must've been an error on my end.

I thought that because it worked in one place, it'll work everywhere. Also didn't know eval can bubble up. Great catch! Thanks for walking me through it.