I am trying to store a list of searches in a lookup table and then pass each search to the map command.
|inputlookup storedsearches.csv | fields + main | map search="search index=myindex sourcetype=mysourcetype $main$"
The storedsearches.csv contains commands such as
|stats count by App and
|stats count by User. The idea is to have Splunk loop thru the base search in the map command and then execute each stats command. I can accomplish this with the
append command but I do not want to worry about max results and search timeouts. Plus I do not want to have to edit the search every time I add or remove a stat command or some other check. I would rather update a lookup table.
This almost works. The problem is how Splunk passes the value to the map command. If there are any spaces or special characters in the lookup table, Splunk automatically adds quotes to the pipeline. If I use a single word with letters or numbers, the quotes are not added.
This is the error I receive.
'index=myindex sourcetype=mysourcetype "|stats count by Application"'
As you can see, the quotes are causing Splunk to not interpret the stats command.
I tried creating a macro and stored the macro name in the lookup table. However, map does not seem to interpret the back-ticks to call the macro.
Any ideas on how to instruct Splunk not to add the quotes?
Wow I literally just posted a search like this in slack chat. Let me find the pastebin code and get back to you.
| inputlookup searches
| map maxsearches=10 search="search
[ stats count
| eval search=\"$search_string$\"
| table search ]"
In your lookup you have the following:
I'm honestly not sure what the stats count does... or if it's even supposed to be there... this is just a fragment of an much larger and older process I havent touched in over 3 years.
Here it is in it's fuller context: http://pastebin.com/8qHv3VqU
The idea was to put different SAP numbers in different indexes based upon what source they came from... it was a huge bandaid to an ingestion issue.
I think the point is you have to escape the quotes with a preceding backslash \
Thanks for the help jkat54. The \ still will not dump the quotes. But you got me and my team thinking. We found a solution, check it out. 🙂
Can you please let me know the solution, we are facing a similar issue
I believe you can't get rid of the quotes. A string value passed to map subsearch get enclosed within double quotes.
We found a way to remove the quotes with a macro, check out the answer. 🙂
A big thanks to my Splunk partner in crime C.F. We were able to solve this issue by calling a macro. The quotes are removed from the lookup returned value when inserted into a macro.
We can successfully add searches to a lookup table and loop thru each search with the map command. Here is how we did it.
First: Create your lookup table with a list of searches to execute. In our case we used lookup editor and named the field "check". Create your csv file however you want.
check, Description index=_internal sourcetype=splunkd component=WatchedFile | stats earliest(_time) as firstime latest(_time) as lasttime count by component log_level, Internal WatchFile Check index=_internal sourcetype=splunkd component=TcpOutputProc | stats earliest(_time) as firstime latest(_time) as lasttime count by component log_level, Internal TcpOutPutProc Check
Second: Create a macro. Give the macro a name with a value of 1. The search will just be a name surrounded by $'s and a single argument. The search defined in the lookup table will be passed to this macro.
Third: Create the search that calls the lookup table and passes the results to the macro with the map command.
The lookup table is called. Only the check field is outputted because there no reason to also pass the description field. The description field is for the admins to remember what the search is looking for.
The map command is called, which includes the macro and the output from the lookup table is inserted into the required variable. Don't forget about the backticks when calling a macro.
|inputlookup internalchecks.csv | fields + check | map search="search `internalcheck($check$)`" maxsearches=100
The Results. The job inspector shows how each search was expanded by the macro.
All my users have to do now for is add searches to the lookup table with out modifying the search. The idea behind this search is to perform validation checks against data in Splunk. Does X field have Y value? Or does X field have the correct length and numeric output? Sometimes a search with large amounts of appends, ANDs, ORs, NOTs can be cumbersome to maintain.
Would you rather maintain a lookup table with a small search or the search below? By looping thru the lookup table, we avoid appending and sub-searches.
index=_internal sourcetype=splunkd component=WatchedFile | stats earliest(_time) as firstime latest(_time) as lasttime count by component log_level | append [search index=_internal sourcetype=splunkd component=TcpOutputProc | stats earliest(_time) as firstime latest(_time) as lasttime count by component log_level]
This is a fantastically easy solution to a very difficult problem. Thank you very very much for posting it, and documenting it so thoroughly!