Splunk Search

Can you create/modify a lookup file via REST API?

a212830
Champion

Hi,

Is it possible to create/modify a lookup file via Splunk's REST API? I don't see anything that addresses this functionality (which, in my mind, is a big hole).

Labels (2)
Tags (4)

4n0m4l1
Engager

If anyone is still looking for a python version for this, it can be found here: https://github.com/M2Curry/push2splunk

woodcock
Esteemed Legend

You can create/modify one via search with |outputlookup and you can post a search via REST but there is no endpoint to create one in core Splunk. If you install Lookup Editor, it creates a REST endpoint that you can the use.

0 Karma

hughkelley
Path Finder

Very clever, @dstaulcu. I expanded on the SPL a bit to include a multi-value field with pipe delimiters.

| stats count as field1 
 | eval field1="host1,1111,A|b;host2,2222,c|d;host3,3333," 
 | eval field1=split(field1,";") 
 | mvexpand field1 
 | rex field=field1 "(?<host>.*),(?<serial>.*),(?<other>.*)" 
 | eval other=split(other,"|") 
 | table host serial other

sloshburch
Ultra Champion

Why not use the makeresults command?

dstaulcu
Builder

didnt know about makeresults at the time. thatd work too!

0 Karma

dstaulcu
Builder

even better.. thanks!

0 Karma

dstaulcu
Builder

You can use the stats command to create fields without event data. Building on that, you can pack structured data into a single field and then leverage split, mvexpand, etc to unpack the data into rows and columns and output results to lookup.

| stats count as field1 
| eval field1="host1,54426859;host2,37203728;host3,96588101" 
| eval field1=split(field1,";") 
| mvexpand field1 
| rex field=field1 "(?<host>.*),(?<serial>.*)" 
| table host serial | outputlookup hostserials.csv

See below for a Powershell code snippet which transforms CSV into lookup table generating SPL, which is then passed to a function which implements the standard REST endpoint for searching. (https://${server}:${port}/services/search/jobs/export")

    $server = "your-server-here"
    $port = "8089"
    $username = "admin"

    $sourcefile = "C:\Development\SplunkCSVtoLookupOverREST\hostserials.csv"
    $content = Import-Csv $sourcefile

    $flattext = Out-Null
    foreach ($item in $content) {
        $thisEntry = "$($item.host),$($item.serial)"
        if ($flattext -eq $null) { $flattext = $thisEntry } else { $flattext += ";$($thisEntry)" }    
    }

    if (!($cred)) { $cred = Get-Credential -Message "enter splunk cred" -UserName $username }

    $thesearch = " | stats count as field1 
    | eval field1=`"${flattext}`"
    | eval field1=split(field1,`";`") 
    | mvexpand field1 
    | rex field=field1 `"(?<host>.*),(?<serial>.*)`" 
    | table host serial | outputlookup hostserials.csv"

    write-host $thesearch
    get-search-results -cred $cred -server $server -port $port -search $thesearch

This technique was successful in creating a 100,000 record lookup table.

paramagurukarth
Builder
0 Karma

sloshburch
Ultra Champion

Careful, that page seems applicable for Splunk 4 and, since it's a wiki, the details may no longer be applicable for current releases. Also, remember that some of those notes expose changing underlying splunk code that might be overwritten during an upgrade (so save your work!).

starcher
SplunkTrust
SplunkTrust

For current versions of Splunk I would recommend using KV store based lookups which can easily be maintained via the REST API.
http://dev.splunk.com/view/webframework-developapps/SP-CAAAEZG

And it has the benefit that if you are using Search Head Clustering the KV Store itself handles the replication of the changes for all nodes.

KSinghK
Loves-to-Learn Lots

I am modifying a kvstore lookup using powershell script and custom api call. if anybody needs that...

 

Tags (1)
0 Karma

a212830
Champion

Thanks. I could do all of this outside Splunk, but I'm looking for something within Splunk (module, or even better, an SPL command) that would let users do it.

0 Karma

sloshburch
Ultra Champion

That kinda implies doing stuff like: https://splunkbase.splunk.com/app/1724 (which Damien mentioned)

0 Karma

a212830
Champion

Thanks, not really what I'm looking for though. Was hoping for something similar to dbquery, where I can create the actual lookup as part of my command, and update it that way as well. Don't want to use a gui to create the lookup (or than the actual spl command), don't want to create it/update it via curl at the OS layer. Want it all to work similar to dbquery, only using REST...

Doesn't sound like it's available (though, I will look at the utility listed below...)

0 Karma

sloshburch
Ultra Champion

An easy button it is you want (said like Yoda). 😉 Yea, looks like nothing currently available. Welcome to create it and post your first app! hint hint. lol

0 Karma

Damien_Dallimor
Ultra Champion

If the lookup file is "staged" on the Splunk instance (ie: you might have SCP'd it up) , you can then use :

Create

http://docs.splunk.com/Documentation/Splunk/6.1.3/RESTAPI/RESTknowledge#POST_data.2Flookup-table-fil...

Modify

http://docs.splunk.com/Documentation/Splunk/6.1.3/RESTAPI/RESTknowledge#POST_data.2Flookup-table-fil...

But you can't remotely upload a new lookup file with these REST endpoints , you'd need to create a Custom REST Endpoint to do this.

This app might interest you : https://apps.splunk.com/app/1724/

Lowell
Super Champion

Can anyone explain why 2 years later there STILL isn't a better answer to this question? I shouldn't have to write a custom endpoint to do something as simple as upload a CSV file. If I have to push it to a staging area first, that's fine. Where's the REST endpoint for that? The UI has supported remote uploads ever since the lookups feature was first introduced. What's the deal? If this feature is being intentionally excluded can someone please explain why?

0 Karma

sloshburch
Ultra Champion

Hey @lowell, do you recall if ever a feature request was made for this? It might have not been addressed simply because of other items with higher customer demand taking the dev resources. If you have a feature request I can make sure a corresponding engineering request is in place thereby tracking this AND validating the customer demand.

0 Karma

Lowell
Super Champion

@SloshBurch, Just sent in an enhancement request as case 448563. Anything you can do to promote would be greatly appreciated. Thanks.

sloshburch
Ultra Champion

Thanks! Found it. Following and making sure a JIRA gets requested.