topic Re: dynamic field value extraction in Dashboards & Visualizations

dynamic field value extraction

klee310 — Mon, 09 Dec 2013 05:05:56 GMT

I'm trying to extract a field-value for comparison - in a dynamic fashion. First let me illustrate the problem with some sample data:

  DataType=2, MaxPower=10, MinPower=3, IdlePower=5
  DataType=3, Open=10, Close=23, HappyHour=15

I have a lookup table with something similar to this:

  DataType,FieldName,ValueMax,ValueMin
  2,MaxPower,100,10
  2,MinPower,10,1
  2,IdlePower,50,1
  3,Open,10,5
  ... etc.

Now my search looks like this:

  index=xxx | lookup LOOKUPFILEX DataType | mvexpand FieldName | ...

I want to map the results into a macro which will perform the actual evaluation, this part is ok. Now I'm trying to write this macro and i'm sort of hitting a wall. For example, the macro might be invoked like this:

  `checkRange("$FieldName$",$ValueMax$,$ValueMin$)`

So from within the macro, given the field name "MaxPower", how do I extract the value (10 in this case) so I can perform comparison with its associated max/min range?

I am currently on the path associated with another question "dynamic field substition"

By the way, the reason I don't want to hard-coding the evaluation (MaxPower, MinPower, IdlePower, Open, etc.) is because - what you see here is just a sample; the actual use-case I am confronted with contains upwards of 100 fields that are subject to change. Therefore modifications must be easily implemented (like via a lookup table)

Re: dynamic field value extraction

klee310 — Mon, 09 Dec 2013 05:10:02 GMT

also, I want to avoid rex if possible since the field DataType, MaxPower, etc. are already automatically extracted by the search-engine; I just need to find a way to reference those values for comparison based on the name of the field dynamically

Re: dynamic field value extraction

gkanapathy — Mon, 09 Dec 2013 05:47:09 GMT

Why can't you just use ValueMin and ValueMax? Isn't the point of the lookup that the max and min are going to be stored in those variables, so you can just them?

Re: dynamic field value extraction

gkanapathy — Mon, 09 Dec 2013 05:48:33 GMT

In other words, why can't you just use if(MaxPower<=ValueMax AND MaxPower >= ValueMin,1,null())?

Re: dynamic field value extraction

klee310 — Mon, 09 Dec 2013 06:12:59 GMT

thanks gkanapathy for the quick response; if I have something like | eval foo=if(MaxPower<=ValueMax AND MaxPower>=ValueMin,1,null())... then MaxPower and MinPower will have to be hard-coded into my macro. Which is not want I want to do. On the other hand, if you are question why I don't use ... eval foo=if($arg1$ <= ValueMax...) well $arg1$ is a string, in this example, it is the string "MaxPower" - so that doesn't work either

Re: dynamic field value extraction

somesoni2 — Mon, 09 Dec 2013 07:01:33 GMT

Your search

index=xxx | lookup LOOKUPFILEX DataType | mvexpand FieldName

Will give all fields present in index=xxx (already extracted by splunk) and fields ValueMin and ValueMax.

Create a macro which will take 4 arguments.

Macro Name: checkRange(4)

Definition:

eval validationResult=if($FieldValue$>=$ValueMin$ AND $FieldValue$<=$ValueMax$,$FieldName$." within range",$FieldName$." out of range")

Arguments : FieldValue,FieldName,ValueMin,ValueMax

Modified search (say validate MinPower field value):

index=xxx | lookup LOOKUPFILEX DataType | mvexpand FieldName |`checkRange(MinPower,"MinPower", ValueMin, ValueMax)`

This should return the field validationResult created in Macro. Obviously, you have to customize this per your requirement, but this should give you fair idea.

Re: dynamic field value extraction

klee310 — Mon, 09 Dec 2013 07:23:54 GMT

thanks somesoni for the reply, but I'm afraid this solution doesn't solve the problem I am having. If I had 100+ FieldName(s), how would I go about invoking this "Macro"? Wouldn't I need 100+ savedsearches, each unique to a FieldName?

Re: dynamic field value extraction

gkanapathy — Mon, 09 Dec 2013 08:43:45 GMT

Nope I don't. Actually I think you can't do this with macros (which is fundamentally one of the differences you get between macros and true functions). I guess my suggestion might be to use a scripted lookup or a custom search command. If you use a scripted lookup, you could combine it with the LOOKUPFILEX lookup. Or you could keep it separate.

Re: dynamic field value extraction

klee310 — Fri, 03 Jan 2014 03:39:51 GMT

actually as it turns out, i did write all 100+ macros - but each is just a simple wrapper which calls a main-macro with the appropriate arguments