topic Re: Newbie map question in Dashboards & Visualizations

Newbie map question

sheltomt — Fri, 19 Jan 2018 19:45:22 GMT

I'm just starting to play with the joys of a Map command, but I can't seem to find the right way to do it. I'm using a test lookup file, and only trying to match one value off it.

Can someone give me some direction, please?

|inputlookup My_Test_Extraction_Lookups 
|table stanza 
|map [| rest splunk_server=local /servicesNS/-/-/data/ui/views 
| rex field=eai:data "(sourcetype=)+(?<sourcetype>\w+)\s" max_match=0
| rename eai:data as code eai:acl.owner as owner eai:acl.app as app 
|stats values(*) as * by sourcetype 
| where sourcetype=$stanza$ 
| table sourcetype title owner app]

Re: Newbie map question

micahkemp — Fri, 19 Jan 2018 20:10:13 GMT

I think you may need to include some additional information about the problem you're running into. Is the search giving you an error, incomplete results, or incorrect results?

Re: Newbie map question

sheltomt — Fri, 19 Jan 2018 20:14:30 GMT

I apologize, it is returning 0 results. I should have ~80 records in the lookup to check against, and I should have about 140ish total records in the end

Re: Newbie map question

cmerriman — Fri, 19 Jan 2018 20:40:08 GMT

your syntax is just a little off. https://docs.splunk.com/Documentation/SplunkCloud/6.6.3/SearchReference/Map

try this:

 |inputlookup My_Test_Extraction_Lookups 
 |table stanza
 |map search="| rest splunk_server=local /servicesNS/-/-/data/ui/views 
 | rex field=eai:data "(sourcetype=)+(?<sourcetype>\w+)\s" max_match=0
 | rename eai:data as code eai:acl.owner as owner eai:acl.app as app 
 |stats values(*) as * by sourcetype
 |search sourcetype=$stanza$
 | table sourcetype title owner app"

Re: Newbie map question

elliotproebstel — Fri, 19 Jan 2018 20:43:21 GMT

In my environment, both syntax structures work - but using the map search="..." structure does require escaping all double-quotes within the search.

Re: Newbie map question

micahkemp — Fri, 19 Jan 2018 20:44:40 GMT

If you use search=" syntax you have to escape any " in your search, like this:

  |inputlookup My_Test_Extraction_Lookups 
  |table stanza
  |map search="| rest splunk_server=local /servicesNS/-/-/data/ui/views 
  | rex field=eai:data \"(sourcetype=)+(?<sourcetype>\w+)\s\" max_match=0
  | rename eai:data as code eai:acl.owner as owner eai:acl.app as app 
  |stats values(*) as * by sourcetype
  |search sourcetype=$stanza$
  | table sourcetype title owner app"

And | map [<search>] is indeed valid.

Re: Newbie map question

elliotproebstel — Fri, 19 Jan 2018 20:57:14 GMT

This is a great example of a search that can be restructured to not use the map command at all.

Why you should avoid using map whenever possible
The map command launches a new search for every line of input that is piped to it. So if the primary search feeding into your map command has 100 lines, then Splunk will launch 100 new searches. (Technically, it will actually only launch 10, unless you use the maxsearches option, because map will default to maxsearches=10. But still.) This is tremendously resource-intensive, so map should be your last option, and you should take great care to see if your searches can be restructured to not use it.

So here's an attempt to restructure your search to not use map:

| rest splunk_server=local /servicesNS/-/-/data/ui/views 
| rex field=eai:data "(sourcetype=)+(?<sourcetype>\w+)\s" max_match=0
| rename eai:data as code eai:acl.owner as owner eai:acl.app as app 
| stats values(*) as * by sourcetype 
| lookup My_Test_Extraction_Lookups stanza AS sourcetype OUTPUT stanza AS found
| where isnotnull(found)
| table sourcetype title owner app

What I've done here is change this into a single pipelined search and used the lookup file to actually perform lookups against the results of that rest call. Since you were only using the stanza value to match against the sourcetype value, it was pretty straightforward to translate the inputlookup command into a lookup call further down the pipeline. I output the stanza value in a new field called found and use it as a flag in the next step to save only the results where the pseudo-flag of found is present.

If this still doesn't give you the results you're looking for, I'm happy to help troubleshoot with some more details.

Re: Newbie map question

sheltomt — Mon, 22 Jan 2018 14:20:04 GMT

Thank you for the explanation and code!

I'll use this for further automation; I did it manually this time around but will definitely use the lookup table as you suggest next time.

Re: Newbie map question

elliotproebstel — Mon, 22 Jan 2018 14:37:36 GMT

Happy to help! When I first discovered the map command, I thought I'd finally found the answer to all my SPL pains, and it took me a while to find out why this magical new command was grinding my searches to a halt. 🙂