Splunk Search
Highlighted

Is there a way to instruct Splunk to not add quotes when passing searches stored in a lookup table to the map command?

Communicator

I am trying to store a list of searches in a lookup table and then pass each search to the map command.

|inputlookup storedsearches.csv | fields + main | map search="search index=myindex sourcetype=mysourcetype $main$"

The storedsearches.csv contains commands such as |stats count by App and |stats count by User. The idea is to have Splunk loop thru the base search in the map command and then execute each stats command. I can accomplish this with the append command but I do not want to worry about max results and search timeouts. Plus I do not want to have to edit the search every time I add or remove a stat command or some other check. I would rather update a lookup table.

This almost works. The problem is how Splunk passes the value to the map command. If there are any spaces or special characters in the lookup table, Splunk automatically adds quotes to the pipeline. If I use a single word with letters or numbers, the quotes are not added.

This is the error I receive.

'index=myindex sourcetype=mysourcetype "|stats count by Application"'

As you can see, the quotes are causing Splunk to not interpret the stats command.

I tried creating a macro and stored the macro name in the lookup table. However, map does not seem to interpret the back-ticks to call the macro.

Any ideas on how to instruct Splunk not to add the quotes?

Highlighted

Re: Is there a way to instruct Splunk to not add quotes when passing searches stored in a lookup table to the map command?

SplunkTrust
SplunkTrust

Wow I literally just posted a search like this in slack chat. Let me find the pastebin code and get back to you.

0 Karma
Highlighted

Re: Is there a way to instruct Splunk to not add quotes when passing searches stored in a lookup table to the map command?

SplunkTrust
SplunkTrust

| inputlookup searches
| map maxsearches=10 search="search
[ stats count
| eval search=\"$search_string$\"
| table search ]"

In your lookup you have the following:
search_string
index=foo sourcetype=bar

I'm honestly not sure what the stats count does... or if it's even supposed to be there... this is just a fragment of an much larger and older process I havent touched in over 3 years.

Here it is in it's fuller context: http://pastebin.com/8qHv3VqU

The idea was to put different SAP numbers in different indexes based upon what source they came from... it was a huge bandaid to an ingestion issue.

Highlighted

Re: Is there a way to instruct Splunk to not add quotes when passing searches stored in a lookup table to the map command?

SplunkTrust
SplunkTrust

I think the point is you have to escape the quotes with a preceding backslash \

0 Karma
Highlighted

Re: Is there a way to instruct Splunk to not add quotes when passing searches stored in a lookup table to the map command?

Communicator

Thanks for the help jkat54. The \ still will not dump the quotes. But you got me and my team thinking. We found a solution, check it out. 🙂

0 Karma
Highlighted

Re: Is there a way to instruct Splunk to not add quotes when passing searches stored in a lookup table to the map command?

New Member

@sjaworski

Can you please let me know the solution, we are facing a similar issue

0 Karma
Highlighted

Re: Is there a way to instruct Splunk to not add quotes when passing searches stored in a lookup table to the map command?

SplunkTrust
SplunkTrust

I believe you can't get rid of the quotes. A string value passed to map subsearch get enclosed within double quotes.

0 Karma
Highlighted

Re: Is there a way to instruct Splunk to not add quotes when passing searches stored in a lookup table to the map command?

Communicator

We found a way to remove the quotes with a macro, check out the answer. 🙂

0 Karma
Highlighted

Re: Is there a way to instruct Splunk to not add quotes when passing searches stored in a lookup table to the map command?

Communicator

A big thanks to my Splunk partner in crime C.F. We were able to solve this issue by calling a macro. The quotes are removed from the lookup returned value when inserted into a macro.

We can successfully add searches to a lookup table and loop thru each search with the map command. Here is how we did it.

First: Create your lookup table with a list of searches to execute. In our case we used lookup editor and named the field "check". Create your csv file however you want.

check, Description
index=_internal sourcetype=splunkd component=WatchedFile | stats earliest(_time) as firstime latest(_time) as lasttime count by component log_level, Internal WatchFile Check
index=_internal sourcetype=splunkd component=TcpOutputProc | stats earliest(_time) as firstime latest(_time) as lasttime count by component log_level, Internal TcpOutPutProc Check

Second: Create a macro. Give the macro a name with a value of 1. The search will just be a name surrounded by $'s and a single argument. The search defined in the lookup table will be passed to this macro.
alt text

Third: Create the search that calls the lookup table and passes the results to the macro with the map command.
The lookup table is called. Only the check field is outputted because there no reason to also pass the description field. The description field is for the admins to remember what the search is looking for.
The map command is called, which includes the macro and the output from the lookup table is inserted into the required variable. Don't forget about the backticks when calling a macro.

|inputlookup internalchecks.csv | fields + check | map search="search `internalcheck($check$)`" maxsearches=100

The Results. The job inspector shows how each search was expanded by the macro.

alt text

All my users have to do now for is add searches to the lookup table with out modifying the search. The idea behind this search is to perform validation checks against data in Splunk. Does X field have Y value? Or does X field have the correct length and numeric output? Sometimes a search with large amounts of appends, ANDs, ORs, NOTs can be cumbersome to maintain.

Would you rather maintain a lookup table with a small search or the search below? By looping thru the lookup table, we avoid appending and sub-searches.

index=_internal sourcetype=splunkd component=WatchedFile | stats earliest(_time) as firstime latest(_time) as lasttime count by component log_level | append [search index=_internal sourcetype=splunkd component=TcpOutputProc | stats earliest(_time) as firstime latest(_time) as lasttime count by component log_level]

View solution in original post

Highlighted

Re: Is there a way to instruct Splunk to not add quotes when passing searches stored in a lookup table to the map command?

Motivator

This is a fantastically easy solution to a very difficult problem. Thank you very very much for posting it, and documenting it so thoroughly!

0 Karma