How to upload updated lookup CSV to Splunk Cloud using REST API WITHOUT using the UI?

[gkiffney]

We're heavy SplunkCloud users and have run into a roadblock. We have a lookup CSV file that needs to be updated daily - slowly changing customer information - but try as I might, I cannot find an automated way to upload these CSVs without using the Web user interface.

The closest thing I can find is

https://<host>:<mPort>/services/data/lookup-table-files/{name}

where the POST method will allow you to "Modify a lookup table file by replacing it with a file from
the upload staging area."

But in SplunkCloud, we don't have access to the upload staging area - we don't have file access at all, as far as I can tell.

How can this be done? I'd like to do this using something simple like curl:

curl -k -u admin:password --form upload=@/home/me/lookup.csv https://mycompany.splunkcloud.com:8089/rest-api-call-to-upload-and-update-existing-csv-lookup

[dmarling]

I documented an answer for this that I believe can be used for cloud customer here: https://answers.splunk.com/answers/694345/how-to-upload-csv-data-file-into-splunk-by-using-r.html?ch...

It requires powershell, but if some enterprising soul wants to port the concepts to another language, it should be possible.

[micahkemp]

I looked into this and found no way to do this via REST, even via undocumented endpoints. I looked at the splunkd_access logs from a timeperiod during which I uploaded a CSV and saw no reference to the upload going through the API. It seems that splunkweb handles the upload and storing the CSV on disk, and then calls a REST endpoint to create the lookup itself using the uploaded file.

Unfortunately I think the answer to this question is simply that it's not possible.

[mbintz]

+1 To this question. I have a lookup table that I'd like to update on a daily basis from a cron job. It would be great if there was a RESTful way to do this.

[gkiffney]

This isn't an answer, more of a workaround. I upload the CSV file, a list of practices, to splunk cloud using the forwarder. In props.conf I have this:

[source::.../lookups/ss_practices.csv]
DATETIME_CONFIG=CURRENT
NO_BINARY_CHECK=true
INDEXED_EXTRACTIONS=CSV
HEADER_FIELD_LINE_NUMBER=1

And I run a scheduled report that looks like this:

index=smrts_maint source="*ss_practices.csv" sourcetype=CSV | table NATIVE_PRACTICE_ID CUSTOMER_NAME STREET CITY STATE_REGION POSTAL_CODE COUNTRY_CODE_A2 COUNTRY_NAME TELEPHONE LAT LON | outputlookup ss_practices.csv

and that updates the lookup table ss_practices.csv. DATETIME_CONFIG = CURRENT is necessary to keep Splunk from trying to guess timestamps for the rows in your csv file if it has no timestamps.

[Micheal_S]

I also took a similar approach based on your answer. I used a scripted input to download the csv I needed and pull it into an index and then your sourcetype for formatting. 

../app/bin/download_csv.sh
Ensure that this file has the right permissions for the splunkd user, I also ensured that it was executable. 

#!/bin/bash

URL="https://www.somesite.com/myfile.csv"

curl -k -s $URL

../app/local/inputs.conf

[script://./bin/download_csv.sh]
disabled = false
interval = * * * * *
index = myindex
sourcetype = mytype_csv

../app/local/props.conf

[mytype_csv]
DATETIME_CONFIG=CURRENT
NO_BINARY_CHECK=true
INDEXED_EXTRACTIONS=CSV
HEADER_FIELD_LINE_NUMBER=1

Then I'll update the csv in a similar fashion to your own with the search. 

[simonnallen]

I am using the Java SDK and would like to be able to add watch lists to Spunk i.e publish IOC's as a lookup that I can the use is queries.

Note I have already implemented the API's to allow me to connect/auth and execute queries. I take these results and analyze the results. This leaves me with a list of known threats. I then want to automatically publish these back to Splunk as a Black Watch List.

Does anyone know how to do this ?

[bizmate]

I also would like a flexible API to upload data, I ll keep investigating but unfortunately for a project I am working on the uploader or doing it by end is not right
I hope they answer with something workable

[clong_]

I also have this question. It doesn't seem to be possible via the API, which is silly considering you can use the GUI to upload lookup tables remotely without having to jump through the staging area hoops. Hopefully the API has or will implement similar capabilities.