upload data directly from a http stream of feeds

[okcerto]

Hi there,

I would like to know if there is an option to upload data directly from a http stream of feeds.

Example:

The "http_slash_slash_url_address" returns a something like this:

{"_origin":"banktrojan","env":{"remote_addr":"212.5.158.188"},"trojanfamily":"Rimecud","data":["\b\f\u00c2\u00832C@y\u001b\u00c2\u00b0@"],"hostn":"lab14","_provider":"lab","trojanproto":"udp","_ts":1381584962,"_geo_env_remote_addr":{"ip":"212.5.158.188","path":"env_remote_addr","country_code":"BG","country_name":"Bulgaria","latitude":43,"longitude":25,"asn":8866,"asn_name":"Bulgarian Telecommunication Company Plc."}}
{"_origin":"banktrojan","env":{"remote_addr":"31.174.17.139"},"trojanfamily":"Rimecud","data":["ƒ2#&´\/np"],"hostn":"lab14","_provider":"lab","trojanproto":"udp","_ts":1381584962,"_geo_env_remote_addr":{"ip":"31.174.17.139","path":"env_remote_addr","country_code":"PL","country_name":"Poland","latitude":52,"longitude":20,"asn":39603,"asn_name":"P4 Sp. z o.o."}}
{"_origin":"banktrojan","env":{"remote_addr":"83.150.82.170"},"trojanfamily":"Rimecud","data":["€K"],"hostn":"lab14","_provider":"lab","trojanproto":"udp","_ts":1381584962,"_geo_env_remote_addr":{"ip":"83.150.82.170","path":"env_remote_addr","country_code":"FI","country_name":"Finland","region":"Southern Finland","city":"Helsinki","latitude":60.1756,"longitude":24.9342,"asn":13276,"asn_name":"Nebula Internet international operations AS"}}
{"_origin":"banktrojan","seen":1381584963,"env":{"remote_addr":"182.178.206.99","path_info":"\/ldr.php","request_method":"POST","http_user_agent":"Mozilla\/4.0"},"trojanfamily":"Zeus","hostn":"lab14","_provider":"lab","_ts":1381584963,"_geo_env_remote_addr":{"ip":"182.178.206.99","path":"env_remote_addr","country_code":"PK","country_name":"Pakistan","region":"Punjab","city":"Lahore","latitude":31.5496,"longitude":74.3436,"asn":45595,"asn_name":"Pakistan Telecom Company Limited"}}
{"_origin":"banktrojan","seen":1381584963,"env":{"remote_addr":"201.29.117.248","path_info":"\/print\/eup.html","request_method":"GET","http_user_agent":"Mozilla\/3.0 (compatible; Indy Library)"},"trojanfamily":"Carufax","hostn":"lab14","_provider":"lab","_ts":1381584963,"_geo_env_remote_addr":{"ip":"201.29.117.248","path":"env_remote_addr","country_code":"BR","country_name":"Brazil","region":"Rio de Janeiro","city":"Rio De Janeiro","latitude":-22.8999,"longitude":-43.2333,"asn":7738,"asn_name":"Telemar Norte Leste S.A."}}
{"dtype":"NginxLog","_origin":"banktrojan","env":{"remote_addr":"2.30.182.107"},"trojanfamily":"W32Expiro","data":["POST owyrohikypa.org HTTP\/1.1"],"_provider":"lab","_ts":1381584962,"_geo_env_remote_addr":{"ip":"2.30.182.107","path":"env_remote_addr","country_code":"GB","country_name":"United Kingdom","region":"London, City of","city":"London","latitude":51.5142,"longitude":-0.093,"asn":12576,"asn_name":"Orange Personal Communications Services"}}

Is it possible to upload it directly to splunk storm?

Thanks

[jpass]

You could create an input script using python or perl that requests the feed. The process might be like this:

1. perl script would request the url and compare a certain field (date or unique id?) to a stored value called "last_date_checked" or "last_id" which you'd need your script to record somewhere after each run. I use perl module "Config::Simple" which allows me to store a value for this purpose. This value is used to track the returned events so you can determine if the incoming events are new or not. If they're new, print them to screen.

2. the script simply prints events to the screen

3. in splunk, add a scripted input and run it at an interval suitable to your needs

http://docs.splunk.com/Documentation/Splunk/6.0/Data/Setupcustominputs

[jpass]

Oh sorry I did not read your request fully. I see you're using Splunk Storm in which case I can't help you.

[bizmate]

Is there an API instead to upload files to an index?

[okcerto]

Is there anybody out there?!?

[okcerto]

Any news about my question?