topic Can you create/modify a lookup file via REST API? in Splunk Search

Can you create/modify a lookup file via REST API?

a212830 — Wed, 02 Dec 2020 04:16:22 GMT

Hi,

Is it possible to create/modify a lookup file via Splunk's REST API? I don't see anything that addresses this functionality (which, in my mind, is a big hole).

Re: Can you create/modify a lookup file via REST API?

Damien_Dallimor — Mon, 25 Aug 2014 04:09:12 GMT

If the lookup file is "staged" on the Splunk instance (ie: you might have SCP'd it up) , you can then use :

Create

http://docs.splunk.com/Documentation/Splunk/6.1.3/RESTAPI/RESTknowledge#POST_data.2Flookup-table-files

Modify

http://docs.splunk.com/Documentation/Splunk/6.1.3/RESTAPI/RESTknowledge#POST_data.2Flookup-table-files.2F.7Bname.7D

But you can't remotely upload a new lookup file with these REST endpoints , you'd need to create a Custom REST Endpoint to do this.

This app might interest you : https://apps.splunk.com/app/1724/

Re: Can you create/modify a lookup file via REST API?

starcher — Thu, 10 Mar 2016 19:51:05 GMT

For current versions of Splunk I would recommend using KV store based lookups which can easily be maintained via the REST API.
http://dev.splunk.com/view/webframework-developapps/SP-CAAAEZG

And it has the benefit that if you are using Search Head Clustering the KV Store itself handles the replication of the changes for all nodes.

Re: Can you create/modify a lookup file via REST API?

a212830 — Mon, 14 Mar 2016 10:54:02 GMT

Thanks. I could do all of this outside Splunk, but I'm looking for something within Splunk (module, or even better, an SPL command) that would let users do it.

Re: Can you create/modify a lookup file via REST API?

sloshburch — Thu, 17 Mar 2016 20:22:46 GMT

That kinda implies doing stuff like: https://splunkbase.splunk.com/app/1724 (which Damien mentioned)

Re: Can you create/modify a lookup file via REST API?

paramagurukarth — Fri, 18 Mar 2016 10:14:30 GMT

Try this..
http://wiki.splunk.com/Community:40GUIDevelopment

Re: Can you create/modify a lookup file via REST API?

a212830 — Fri, 18 Mar 2016 12:11:05 GMT

Thanks, not really what I'm looking for though. Was hoping for something similar to dbquery, where I can create the actual lookup as part of my command, and update it that way as well. Don't want to use a gui to create the lookup (or than the actual spl command), don't want to create it/update it via curl at the OS layer. Want it all to work similar to dbquery, only using REST...

Doesn't sound like it's available (though, I will look at the utility listed below...)

Re: Can you create/modify a lookup file via REST API?

sloshburch — Mon, 21 Mar 2016 14:47:04 GMT

Careful, that page seems applicable for Splunk 4 and, since it's a wiki, the details may no longer be applicable for current releases. Also, remember that some of those notes expose changing underlying splunk code that might be overwritten during an upgrade (so save your work!).

Re: Can you create/modify a lookup file via REST API?

sloshburch — Mon, 21 Mar 2016 14:48:12 GMT

An easy button it is you want (said like Yoda). 😉 Yea, looks like nothing currently available. Welcome to create it and post your first app! hint hint. lol

Re: Can you create/modify a lookup file via REST API?

Lowell — Thu, 09 Feb 2017 21:56:00 GMT

Can anyone explain why 2 years later there STILL isn't a better answer to this question? I shouldn't have to write a custom endpoint to do something as simple as upload a CSV file. If I have to push it to a staging area first, that's fine. Where's the REST endpoint for that? The UI has supported remote uploads ever since the lookups feature was first introduced. What's the deal? If this feature is being intentionally excluded can someone please explain why?

Re: Can you create/modify a lookup file via REST API?

sloshburch — Fri, 10 Feb 2017 13:28:30 GMT

Hey @lowell, do you recall if ever a feature request was made for this? It might have not been addressed simply because of other items with higher customer demand taking the dev resources. If you have a feature request I can make sure a corresponding engineering request is in place thereby tracking this AND validating the customer demand.

Re: Can you create/modify a lookup file via REST API?

Lowell — Fri, 10 Feb 2017 15:22:46 GMT

I do not have an official feature request in at this time. I was just surprised to see a few similar questions posted here, but no real movement in a few years. The additional complexity I haven't noted yet is that I need a solution that works with Search Head Clustering. I need to be able to consistently programmatically deploy a lookup file to all the members of the cluster. Ideally, I'd be able to not only push a new lookup, but cleanly replace an existing one.

I'll work with my client to get an enhancement request created.

Re: Can you create/modify a lookup file via REST API?

starcher — Fri, 10 Feb 2017 15:27:55 GMT

Yeah without ERs just because it's in Answers doesn't mean it will work its way up the priority chain.
The best solution to do it programmatically is use KVStore lookups which can be handled via rest API.

You can see it mentioned in conf 2016 talk:
https://conf.splunk.com/sessions/2016-sessions.html#
Shop Smart at the KV Store: Best Value Tricks from the Splunk KV Store and REST API

Re: Can you create/modify a lookup file via REST API?

Lowell — Fri, 10 Feb 2017 15:37:34 GMT

Understood. My primary use case is just updating simple (typically 100 lines or less, often less than 1 KB) lookup tables. And mostly I'm looking to do this in just TAs where I want to be able to dictate the exact content of the entire table, maintain them through version control, and so on. I agree that there are lots of other places where KVstore is the ideal solution.

Re: Can you create/modify a lookup file via REST API?

Lowell — Fri, 10 Feb 2017 20:16:21 GMT

@SloshBurch, Just sent in an enhancement request as case 448563. Anything you can do to promote would be greatly appreciated. Thanks.

Re: Can you create/modify a lookup file via REST API?

sloshburch — Mon, 13 Feb 2017 13:35:38 GMT

Thanks! Found it. Following and making sure a JIRA gets requested.

Re: Can you create/modify a lookup file via REST API?

sloshburch — Mon, 13 Feb 2017 22:16:00 GMT

Sanity Check: Are we all on the same page that lookups stay in sync in a SHC when used with generated with outputlookup, but not outputcsv. Right? Are we saying that when using the upload they do NOT stay in sync?

Re: Can you create/modify a lookup file via REST API?

Lowell — Tue, 14 Feb 2017 18:04:56 GMT

I've only been looking at outputlooup because (1) I need an actual lookup, not just stored search results, and (2) The docs say that outputcsv isn't supported on an SHC (not surprising)

I'm not aware of any issues with uploaded lookup tables. My complaint is that you can't upload it via splunkd (REST) directly, you have to do it via the UI. Which is less ideal from a programatic perspective.

Re: Can you create/modify a lookup file via REST API?

dstaulcu — Thu, 01 Jun 2017 01:45:05 GMT

You can use the stats command to create fields without event data. Building on that, you can pack structured data into a single field and then leverage split, mvexpand, etc to unpack the data into rows and columns and output results to lookup.

| stats count as field1 
| eval field1="host1,54426859;host2,37203728;host3,96588101" 
| eval field1=split(field1,";") 
| mvexpand field1 
| rex field=field1 "(?<host>.*),(?<serial>.*)" 
| table host serial | outputlookup hostserials.csv

See below for a Powershell code snippet which transforms CSV into lookup table generating SPL, which is then passed to a function which implements the standard REST endpoint for searching. (https://${server}:${port}/services/search/jobs/export")

    $server = "your-server-here"
    $port = "8089"
    $username = "admin"

    $sourcefile = "C:\Development\SplunkCSVtoLookupOverREST\hostserials.csv"
    $content = Import-Csv $sourcefile

    $flattext = Out-Null
    foreach ($item in $content) {
        $thisEntry = "$($item.host),$($item.serial)"
        if ($flattext -eq $null) { $flattext = $thisEntry } else { $flattext += ";$($thisEntry)" }    
    }

    if (!($cred)) { $cred = Get-Credential -Message "enter splunk cred" -UserName $username }

    $thesearch = " | stats count as field1 
    | eval field1=`"${flattext}`"
    | eval field1=split(field1,`";`") 
    | mvexpand field1 
    | rex field=field1 `"(?<host>.*),(?<serial>.*)`" 
    | table host serial | outputlookup hostserials.csv"

    write-host $thesearch
    get-search-results -cred $cred -server $server -port $port -search $thesearch

This technique was successful in creating a 100,000 record lookup table.

Re: Can you create/modify a lookup file via REST API?

harry2007gsp — Wed, 01 Aug 2018 21:23:00 GMT

Hi guys,
Can we push lookup table data from outside database(mongoDb lookukp collection) to splunk with splunk python sdk?

We have been pushing normal data to splunk with the help of third party JDBC unity drivers but now planning to push it with python splunk sdk. This case is possible and we know how to do it.

Problem is how can we push lookup data to splunk lookup tables instead of indexes.