Can you create/modify a lookup file via REST API?

[a212830]

Hi,

Is it possible to create/modify a lookup file via Splunk's REST API? I don't see anything that addresses this functionality (which, in my mind, is a big hole).

[denissotoacc]

It's a pretty old question, but I managed to create lookup csv files using the REST API by running a search through the API.

Let's suppose you need to create a lookup file inside "my_app", named "my_lookup.csv" with fields "myfield1,myfield2,myfield3":
The CURL might be something like this:

curl -k -u admin:pass https://localhost:8089/servicesNS/admin/my_app/search/jobs search="search | makeresults | eval field1=\"\",field2=\"\",field3=\"\" | table field1 field2 field3 | outputlookup my_lookup.csv create_context=app"

[mthcht]

Hi i made these scripts to update or upload lookups on splunk using lookup-editor endpoint 

It can save the content of lookup(s) from splunk, add new fields and values or merge files

[PaulPanther]

Hi @mthcht thanks for the smart solution! I'm trying to create a lookup via CURL 

curl -X POST -v -k -u admin:pass https://localhost:8089/services/data/lookup_edit/lookup_contents -d 'namespace=search' -d 'lookup_file=test' -d 'lookup_type=csv'

 but I'm getting the response "Unable to save the lookup". I'm using Splunk 9.0.3 standalone. Do you have any idea why the lookup isn't saved? I just need an empty lookup file.

[yeahnah]

Thought I'd add to this post, in regards to using a curl command to push a lookup file to a Splunk instance, as other Splunk users may find it useful.  It's not a replacement for @mthcht excellent python scripts but it is often easy to use curl commands when testing and validating things.  

Here's a worked example that creates a simple lookup file (tested against Cloud stack and lookup editor v4.0.4) ...

curl -sk --request POST https://localhost:8089/services/data/lookup_edit/lookup_contents \
 -H "Authorization: Splunk $MYTOKEN" \
 -H "Content-Type: application/x-www-form-urlencoded" \
 -d timeout=10 \
 -d namespace=search \
 -d lookup_file=lookupfilename.csv  \
 -d contents=[[\"field1\",\"field2\"],[\"value1\",\"value2\"]] \
 -d owner=nobody  # n.b. owner is only needed when creating new lookups - a 'user' name creates the new lookup file with private permissions, whereas 'nobody' results in it being shared globally

Note, the 'contents' format must be a 2D JSON array.

To make this easier, 'contents' can also be added via a file, like this ...

$ cat <<EOF > myLocalLookup.json
contents=[["field1","field2"],["value1","value2"]]
EOF

$ curl -sk --request POST https://localhost:8089/services/data/lookup_edit/lookup_contents \
 -H "Authorization: Splunk $MYTOKEN" \
 -H "Content-Type: application/x-www-form-urlencoded" \
 -d timeout=10 \
 -d namespace=search \
 -d lookup_file=lookupfilename.csv  \
 -d @myLocalLookup.json \
 -d owner=nobody

Now, to really make this useful, existing CSV file's need to be formatted as JSON. There are multiple ways this could be done, but here is a simple python oneliner (*nix tested) that reads in a CSV file on stdin and outputs it as JSON.

(python -c $'import sys;import csv;import json;\nwith sys.stdin as f: csv_array=list(csv.reader(f)); print("contents="+json.dumps(csv_array))' > mylocalLookup.json) < myLocalLookup.csv 

Hopefully. others may find this useful too.

[burwell]

Hi. Do you have access to the logs? Is there a permission error?

[PaulPanther]

@burwell  Don't ask me why I wasn't able to review the internal logs in the right way last Friday. Getting following error:

02/13/2023 11:17:34 AM ERROR Failed to save lookup: details=expected string or bytes-like object
Traceback (most recent call last):
  File "C:\Program Files\Splunk\etc\apps\lookup_editor\bin\lookup_editor_rest_handler.py", line 254, in post_lookup_contents
    request_info.session_key, request_info.user)
  File "C:\Program Files\Splunk\etc\apps\lookup_editor\bin\lookup_editor\__init__.py", line 335, in update
    if not is_file_name_valid(lookup_file):
  File "C:\Program Files\Splunk\etc\apps\lookup_editor\bin\lookup_editor\shortcuts.py", line 129, in is_file_name_valid
    if not allowed_path.match(lookup_file):
TypeError: expected string or bytes-like object

I tried to add additional parameter -d  'contents={"testA":"blabla"}' but still getting the same error message

[burwell]

Hi. You probably want to examine the script that was posted to upload the csv. The script doesn't just do a rest call with the file it loops through and reads the contents in python and then uploads a big string

https://github.com/mthcht/lookup-editor_scripts/blob/main/upload_lookups_to_splunk.py#L39-L47

[burwell]

Thanks @mthcht I tested this in a single search head AND on a search head cluster and it worked in both cases. Thanks for publishing it!

[4n0m4l1]

If anyone is still looking for a python version for this, it can be found here: https://github.com/M2Curry/push2splunk

[hettervik]

Thanks for sharing this, works great! Could also be added in the readme that it's possible to change the destination for the lookups on the target server by changing the following lines in myvariables.py.

From:

SplunkLookupURL = "/services/properties/lookups"
SplunkSearchURL = "/services/search/jobs"

To:

SplunkLookupURL = "/servicesNS/-/<yourapp>/properties/lookups"
SplunkSearchURL = "/servicesNS/-/<yourapp>/search/jobs"

[woodcock]

You can create/modify one via search with |outputlookup and you can post a search via REST but there is no endpoint to create one in core Splunk. If you install Lookup Editor, it creates a REST endpoint that you can the use.

[hughkelley]

Very clever, @dstaulcu. I expanded on the SPL a bit to include a multi-value field with pipe delimiters.

| stats count as field1 
 | eval field1="host1,1111,A|b;host2,2222,c|d;host3,3333," 
 | eval field1=split(field1,";") 
 | mvexpand field1 
 | rex field=field1 "(?<host>.*),(?<serial>.*),(?<other>.*)" 
 | eval other=split(other,"|") 
 | table host serial other

[sloshburch]

Why not use the makeresults command?

[dstaulcu]

didnt know about makeresults at the time. thatd work too!

[dstaulcu]

even better.. thanks!

[dstaulcu]

You can use the stats command to create fields without event data. Building on that, you can pack structured data into a single field and then leverage split, mvexpand, etc to unpack the data into rows and columns and output results to lookup.

| stats count as field1 
| eval field1="host1,54426859;host2,37203728;host3,96588101" 
| eval field1=split(field1,";") 
| mvexpand field1 
| rex field=field1 "(?<host>.*),(?<serial>.*)" 
| table host serial | outputlookup hostserials.csv

See below for a Powershell code snippet which transforms CSV into lookup table generating SPL, which is then passed to a function which implements the standard REST endpoint for searching. (https://${server}:${port}/services/search/jobs/export")

    $server = "your-server-here"
    $port = "8089"
    $username = "admin"

    $sourcefile = "C:\Development\SplunkCSVtoLookupOverREST\hostserials.csv"
    $content = Import-Csv $sourcefile

    $flattext = Out-Null
    foreach ($item in $content) {
        $thisEntry = "$($item.host),$($item.serial)"
        if ($flattext -eq $null) { $flattext = $thisEntry } else { $flattext += ";$($thisEntry)" }    
    }

    if (!($cred)) { $cred = Get-Credential -Message "enter splunk cred" -UserName $username }

    $thesearch = " | stats count as field1 
    | eval field1=`"${flattext}`"
    | eval field1=split(field1,`";`") 
    | mvexpand field1 
    | rex field=field1 `"(?<host>.*),(?<serial>.*)`" 
    | table host serial | outputlookup hostserials.csv"

    write-host $thesearch
    get-search-results -cred $cred -server $server -port $port -search $thesearch

This technique was successful in creating a 100,000 record lookup table.

[paramagurukarth]

Try this..
http://wiki.splunk.com/Community:40GUIDevelopment

[sloshburch]

Careful, that page seems applicable for Splunk 4 and, since it's a wiki, the details may no longer be applicable for current releases. Also, remember that some of those notes expose changing underlying splunk code that might be overwritten during an upgrade (so save your work!).

[starcher]

For current versions of Splunk I would recommend using KV store based lookups which can easily be maintained via the REST API.
http://dev.splunk.com/view/webframework-developapps/SP-CAAAEZG

And it has the benefit that if you are using Search Head Clustering the KV Store itself handles the replication of the changes for all nodes.

[KSinghK]

I am modifying a kvstore lookup using powershell script and custom api call. if anybody needs that...