What are my options for loading external data to Splunk lookups if I cannot have access to the server

[MonkeyK]

Lets say that I periodically get threat data in the forum of reports that contain URLs and IP addresses. I parse these reports and build a list of URLs and IP addresses that I want to add to a lookup table in Splunk so that I can match on the lookup table values in a search.

Is there an efficient way for me to do this if I cannon have access to the server? All of the examples that I see say put a .csv on the server or collect the data via a web request. Can I just dump a bunch of comma separated values on a search and use that to load a csv?

something like "url1","url2","url3"....|outputlookup mybadurllist.csv

[MonkeyK · Accepted solution]

Just found the Lookup File Editor for Splunk Enterprise App.
https://splunkbase.splunk.com/app/1724/#/overview

Looks like this app would let me edit in place similar to a very simple spreadsheet or reimport a csv/kvstore over the existing data. With this I could maintain a CSV of relevant indicators that would be used as search criteria.

If anyone else has the same problem take a look at that app.

[MonkeyK]

Wow! Thank you for pointing that out. We had a Splunk consultant here who could not tell me how to deal with threat intel from sources that I had to parse.
That said, the process outlined does seem cumbersome. I'll read through the topic and see what alternatives it may offer.

[smoir_splunk]

Yes, it's certainly cumbersome to create a lookup from scratch. Depending on the data, you could use a STIX or OpenIOC editor to convert to one of those types. This process will also get easier in the future 🙂

[MonkeyK]

One more think that I noticed about Lookup File Editor is that I can paste into it. The App will automatically increase the number of rows to accommodate the pasted data.

The ES table editor does not do this.

[smoir_splunk]

If you do have Splunk Enterprise Security, this app's functionality is duplicated in ES with local lookup files.

Configure > Lists and Lookups and any of the lookup files named "Local * Intel", e.g., "Local File Intel" are available to edit using this editor.

[MonkeyK]

That is a good point smoir, but I the App does one more thing. The file editor app also allows me to import a csv/kvstore, which I see as a pretty good productivity enhancement.
My concern is loading a decent chunk of data (say 100-200 indicators). If I could only use editing in place, this might take a very long time. But if I can process my indicators to a CSV and then load that directly, I think that I could save a lot of time and avoid typographical errors.

[smoir_splunk]

Absolutely true!

If you're editing a CSV file and then uploading it to Splunk, Enterprise Security also has a workflow to support that.

http://docs.splunk.com/Documentation/ES/4.5.1/User/Configureblocklists#Add_threat_data_manually covers the "edit-in-place" lookup method that I mentioned earlier.

http://docs.splunk.com/Documentation/ES/4.5.1/User/Configureblocklists#Adding_a_custom_source covers adding a lookup file to ES, for example, a CSV file.

The lookup editor app, however, also does this and is a great option for people without Enterprise Security!

[bmacias84]

What is your external data source? Internally we a similar issue, so built a command called reread (short for remote read) which is installed on our SHC. It was something I wrote quick an dirty for a few teams and I haven't documented, but it basically will perform a get to an api or url containing CSV or JSON data. Been meaning to polish it for http://apps.splunk.com but you can get the source from github

| reread csv http://someur.com/yourcsv.csv

or

| reread json http://someurl.com/jsonApiEndPoint

With this could would just tack on the something like this;

... | table url ip | outputlookup mybadeurllist.csv

Note: There are some other arguments which allow for timestamp pattern matching and line breaking, but you should have to worry about that.

[MonkeyK]

So then it sounds like your teams write to a web server that Splunk is authorized to access? I don't even have that.
I guess I could write an app to publish files to a web server...
But in that case, maybe I should just write to a database and have Splunk get my bad list from a database table...

Or a dump all values into an existing ES lookup table as a multivalue field

Seems like something that should be easier to do