Feature Request: Support GET and name-value pairs in automatic external looups

[dpadams]

I've been struggling a bit with external lookups. It's potentially a fantastically useful feature, but I've been hampered largely by a lack of Python knowledge and a shortage of good examples. (The docs are decent and I have found some examples, but not enough for me with my limited knowledge of Python and Splunk.) I've got nothing against learning Python, but also don't have the time to do it.

It occurred to me that a nice feature would be to add URL-based lookups as an external lookup system. Then, you could integrate any existing service that deals with HTTP name-value pairs. If you need to write a new back-end service and have a Web/application server, you can use any language you like to make it all work. Again, nothing against Python apart from a shortage of time.

Hopefully, this feature would make external lookups accessible to a broader range of people than they are today.

I've written up a fake props.conf and transforms.conf (below) to show how such a feature might work on the configuration side. I'm not quite sure how to submit feature requests for Splunk. I submitted an 'enhancement request' but, without a license (still in the eval phase), I'm not sure if my submissions get any attention (18 days and counting on my one tech support question.) So, I'm posting the idea here for comment and tagging it 'feature-request'.

Thanks for any suggestions or comments. If I've missed an existing feature that will do what I need, I love to learn about it.

-- Thanks

props.conf sample below:

[application_log]
# Take application_log to be a custom souretype.
lookup_table = user_market_lookup appuser_uuid OUTPUT market_id, market_name

transforms.conf sample below:

[user_market_lookup]

# The external URL is specified below.
# ID is the name that will reach the exteranl Web/app server.
# appuser_uuid is a Splunk field name and is filled in automatically
# when this script is invoked by Splunk as an external lookup.
# Note: It's easy enough to put HTTP Basic 'credentials' into a URL
# but some people might want additional security. I don't need it myself
# at the moment. It would be good to support SSL in the connection.
external_url = https://www.example.com/GetMarketDataFromUserUUID?ID=appuser_uuid

# I supposed you could support POST as well as GET
external_type = get

# The Web/app server returns name-value pairs, like
# market_id=5&market_name=North%20Dakota
# Splunk should convert these into fields using the
# names specified in the fields_list and extract
# the matching values from the Web results. So:
# market_id = 5
# market_name = North Dakota
fields_list = market_id, market_name

# Often enough, a log line with have no entry or a value such as - 
# instead of a meaningful field value. Splunk recognizes this 
# in search results as it is. It would be nice if the external
# lookup could be short-circuited to avoid needless calls to the
# Web/app server. You could have a default value returned without
# making the external call. The default string could be a "",
# a meaningful default, an error message or any other string.
# Some nice optional directives for this could go like so:
result_if_input_equals_none = ""
result_if_input_is_empty = ""

# I'm happy with URL-encoded name-value pairs but others
# might want JSON or XML. You could have an optional
# format declaration with constants like
# XML, JSON, or HTTP
name_value_format = http 

[woodcock]

You roughly describing threatwatchlist:

https://splunkbase.splunk.com/app/635/