Splunk Enterprise Security

What are my options for loading external data to Splunk lookups if I cannot have access to the server

MonkeyK
Builder

Lets say that I periodically get threat data in the forum of reports that contain URLs and IP addresses. I parse these reports and build a list of URLs and IP addresses that I want to add to a lookup table in Splunk so that I can match on the lookup table values in a search.

Is there an efficient way for me to do this if I cannon have access to the server? All of the examples that I see say put a .csv on the server or collect the data via a web request. Can I just dump a bunch of comma separated values on a search and use that to load a csv?

something like "url1","url2","url3"....|outputlookup mybadurllist.csv

0 Karma
1 Solution

MonkeyK
Builder

Just found the Lookup File Editor for Splunk Enterprise App.
https://splunkbase.splunk.com/app/1724/#/overview

Looks like this app would let me edit in place similar to a very simple spreadsheet or reimport a csv/kvstore over the existing data. With this I could maintain a CSV of relevant indicators that would be used as search criteria.

If anyone else has the same problem take a look at that app.

View solution in original post

0 Karma

MonkeyK
Builder

Just found the Lookup File Editor for Splunk Enterprise App.
https://splunkbase.splunk.com/app/1724/#/overview

Looks like this app would let me edit in place similar to a very simple spreadsheet or reimport a csv/kvstore over the existing data. With this I could maintain a CSV of relevant indicators that would be used as search criteria.

If anyone else has the same problem take a look at that app.

View solution in original post

0 Karma

MonkeyK
Builder

Wow! Thank you for pointing that out. We had a Splunk consultant here who could not tell me how to deal with threat intel from sources that I had to parse.
That said, the process outlined does seem cumbersome. I'll read through the topic and see what alternatives it may offer.

0 Karma

smoir_splunk
Splunk Employee
Splunk Employee

Yes, it's certainly cumbersome to create a lookup from scratch. Depending on the data, you could use a STIX or OpenIOC editor to convert to one of those types. This process will also get easier in the future 🙂

0 Karma

MonkeyK
Builder

One more think that I noticed about Lookup File Editor is that I can paste into it. The App will automatically increase the number of rows to accommodate the pasted data.

The ES table editor does not do this.

0 Karma

smoir_splunk
Splunk Employee
Splunk Employee

If you do have Splunk Enterprise Security, this app's functionality is duplicated in ES with local lookup files.

Configure > Lists and Lookups and any of the lookup files named "Local * Intel", e.g., "Local File Intel" are available to edit using this editor.

0 Karma

MonkeyK
Builder

That is a good point smoir, but I the App does one more thing. The file editor app also allows me to import a csv/kvstore, which I see as a pretty good productivity enhancement.
My concern is loading a decent chunk of data (say 100-200 indicators). If I could only use editing in place, this might take a very long time. But if I can process my indicators to a CSV and then load that directly, I think that I could save a lot of time and avoid typographical errors.

0 Karma

smoir_splunk
Splunk Employee
Splunk Employee

Absolutely true!

If you're editing a CSV file and then uploading it to Splunk, Enterprise Security also has a workflow to support that.

http://docs.splunk.com/Documentation/ES/4.5.1/User/Configureblocklists#Add_threat_data_manually covers the "edit-in-place" lookup method that I mentioned earlier.

http://docs.splunk.com/Documentation/ES/4.5.1/User/Configureblocklists#Adding_a_custom_source covers adding a lookup file to ES, for example, a CSV file.

The lookup editor app, however, also does this and is a great option for people without Enterprise Security!

bmacias84
Champion

What is your external data source? Internally we a similar issue, so built a command called reread (short for remote read) which is installed on our SHC. It was something I wrote quick an dirty for a few teams and I haven't documented, but it basically will perform a get to an api or url containing CSV or JSON data. Been meaning to polish it for http://apps.splunk.com but you can get the source from github

| reread csv http://someur.com/yourcsv.csv

or

| reread json http://someurl.com/jsonApiEndPoint

With this could would just tack on the something like this;

... | table url ip | outputlookup mybadeurllist.csv

Note: There are some other arguments which allow for timestamp pattern matching and line breaking, but you should have to worry about that.

MonkeyK
Builder

So then it sounds like your teams write to a web server that Splunk is authorized to access? I don't even have that.
I guess I could write an app to publish files to a web server...
But in that case, maybe I should just write to a database and have Splunk get my bad list from a database table...

Or a dump all values into an existing ES lookup table as a multivalue field

Seems like something that should be easier to do

0 Karma