topic Re: How to upload updated lookup CSV to Splunk Cloud using REST API WITHOUT using the UI? in Splunk Search

How to upload updated lookup CSV to Splunk Cloud using REST API WITHOUT using the UI?

gkiffney — Wed, 02 Dec 2015 17:41:13 GMT

We're heavy SplunkCloud users and have run into a roadblock. We have a lookup CSV file that needs to be updated daily - slowly changing customer information - but try as I might, I cannot find an automated way to upload these CSVs without using the Web user interface.

The closest thing I can find is

https://<host>:<mPort>/services/data/lookup-table-files/{name}

where the POST method will allow you to "Modify a lookup table file by replacing it with a file from
the upload staging area."

But in SplunkCloud, we don't have access to the upload staging area - we don't have file access at all, as far as I can tell.

How can this be done? I'd like to do this using something simple like curl:

curl -k -u admin:password --form upload=@/home/me/lookup.csv https://mycompany.splunkcloud.com:8089/rest-api-call-to-upload-and-update-existing-csv-lookup

Re: How to upload updated lookup CSV to Splunk Cloud using REST API WITHOUT using the UI?

clong_ — Wed, 23 Dec 2015 23:33:03 GMT

I also have this question. It doesn't seem to be possible via the API, which is silly considering you can use the GUI to upload lookup tables remotely without having to jump through the staging area hoops. Hopefully the API has or will implement similar capabilities.

Re: How to upload updated lookup CSV to Splunk Cloud using REST API WITHOUT using the UI?

bizmate — Wed, 02 Mar 2016 22:12:32 GMT

I also would like a flexible API to upload data, I ll keep investigating but unfortunately for a project I am working on the uploader or doing it by end is not right
I hope they answer with something workable

Re: How to upload updated lookup CSV to Splunk Cloud using REST API WITHOUT using the UI?

simonnallen — Tue, 08 Mar 2016 23:46:13 GMT

I am using the Java SDK and would like to be able to add watch lists to Spunk i.e publish IOC's as a lookup that I can the use is queries.

Note I have already implemented the API's to allow me to connect/auth and execute queries. I take these results and analyze the results. This leaves me with a list of known threats. I then want to automatically publish these back to Splunk as a Black Watch List.

Does anyone know how to do this ?

Re: How to upload updated lookup CSV to Splunk Cloud using REST API WITHOUT using the UI?

gkiffney — Tue, 29 Sep 2020 09:01:34 GMT

This isn't an answer, more of a workaround. I upload the CSV file, a list of practices, to splunk cloud using the forwarder. In props.conf I have this:

[source::.../lookups/ss_practices.csv]
DATETIME_CONFIG=CURRENT
NO_BINARY_CHECK=true
INDEXED_EXTRACTIONS=CSV
HEADER_FIELD_LINE_NUMBER=1

And I run a scheduled report that looks like this:

index=smrts_maint source="*ss_practices.csv" sourcetype=CSV | table NATIVE_PRACTICE_ID CUSTOMER_NAME STREET CITY STATE_REGION POSTAL_CODE COUNTRY_CODE_A2 COUNTRY_NAME TELEPHONE LAT LON | outputlookup ss_practices.csv

and that updates the lookup table ss_practices.csv. DATETIME_CONFIG = CURRENT is necessary to keep Splunk from trying to guess timestamps for the rows in your csv file if it has no timestamps.

Re: How to upload updated lookup CSV to Splunk Cloud using REST API WITHOUT using the UI?

mbintz — Wed, 09 Mar 2016 15:07:55 GMT

+1 To this question. I have a lookup table that I'd like to update on a daily basis from a cron job. It would be great if there was a RESTful way to do this.

Re: How to upload updated lookup CSV to Splunk Cloud using REST API WITHOUT using the UI?

micahkemp — Thu, 21 Dec 2017 00:51:05 GMT

I looked into this and found no way to do this via REST, even via undocumented endpoints. I looked at the splunkd_access logs from a timeperiod during which I uploaded a CSV and saw no reference to the upload going through the API. It seems that splunkweb handles the upload and storing the CSV on disk, and then calls a REST endpoint to create the lookup itself using the uploaded file.

Unfortunately I think the answer to this question is simply that it's not possible.

Re: How to upload updated lookup CSV to Splunk Cloud using REST API WITHOUT using the UI?

dmarling — Tue, 14 Apr 2020 15:57:17 GMT

I documented an answer for this that I believe can be used for cloud customer here: https://answers.splunk.com/answers/694345/how-to-upload-csv-data-file-into-splunk-by-using-r.html?childToView=816912#answer-816912

It requires powershell, but if some enterprising soul wants to port the concepts to another language, it should be possible.

Re: How to upload updated lookup CSV to Splunk Cloud using REST API WITHOUT using the UI?

Micheal_S — Thu, 18 May 2023 17:19:02 GMT

I also took a similar approach based on your answer. I used a scripted input to download the csv I needed and pull it into an index and then your sourcetype for formatting.

../app/bin/download_csv.sh
Ensure that this file has the right permissions for the splunkd user, I also ensured that it was executable.

#!/bin/bash URL="https://www.somesite.com/myfile.csv" curl -k -s $URL

../app/local/inputs.conf

[script://./bin/download_csv.sh] disabled = false interval = * * * * * index = myindex sourcetype = mytype_csv

../app/local/props.conf

[mytype_csv] DATETIME_CONFIG=CURRENT NO_BINARY_CHECK=true INDEXED_EXTRACTIONS=CSV HEADER_FIELD_LINE_NUMBER=1

Then I'll update the csv in a similar fashion to your own with the search.