Getting Data In

upload data directly from a http stream of feeds

okcerto
New Member

Hi there,

I would like to know if there is an option to upload data directly from a http stream of feeds.

Example:

The "http_slash_slash_url_address" returns a something like this:

{"_origin":"banktrojan","env":{"remote_addr":"212.5.158.188"},"trojanfamily":"Rimecud","data":["\b\f\u00c2\u00832C@y\u001b\u00c2\u00b0@"],"hostn":"lab14","_provider":"lab","trojanproto":"udp","_ts":1381584962,"_geo_env_remote_addr":{"ip":"212.5.158.188","path":"env_remote_addr","country_code":"BG","country_name":"Bulgaria","latitude":43,"longitude":25,"asn":8866,"asn_name":"Bulgarian Telecommunication Company Plc."}}
{"_origin":"banktrojan","env":{"remote_addr":"31.174.17.139"},"trojanfamily":"Rimecud","data":["ƒ2#&´\/np"],"hostn":"lab14","_provider":"lab","trojanproto":"udp","_ts":1381584962,"_geo_env_remote_addr":{"ip":"31.174.17.139","path":"env_remote_addr","country_code":"PL","country_name":"Poland","latitude":52,"longitude":20,"asn":39603,"asn_name":"P4 Sp. z o.o."}}
{"_origin":"banktrojan","env":{"remote_addr":"83.150.82.170"},"trojanfamily":"Rimecud","data":["€K"],"hostn":"lab14","_provider":"lab","trojanproto":"udp","_ts":1381584962,"_geo_env_remote_addr":{"ip":"83.150.82.170","path":"env_remote_addr","country_code":"FI","country_name":"Finland","region":"Southern Finland","city":"Helsinki","latitude":60.1756,"longitude":24.9342,"asn":13276,"asn_name":"Nebula Internet international operations AS"}}
{"_origin":"banktrojan","seen":1381584963,"env":{"remote_addr":"182.178.206.99","path_info":"\/ldr.php","request_method":"POST","http_user_agent":"Mozilla\/4.0"},"trojanfamily":"Zeus","hostn":"lab14","_provider":"lab","_ts":1381584963,"_geo_env_remote_addr":{"ip":"182.178.206.99","path":"env_remote_addr","country_code":"PK","country_name":"Pakistan","region":"Punjab","city":"Lahore","latitude":31.5496,"longitude":74.3436,"asn":45595,"asn_name":"Pakistan Telecom Company Limited"}}
{"_origin":"banktrojan","seen":1381584963,"env":{"remote_addr":"201.29.117.248","path_info":"\/print\/eup.html","request_method":"GET","http_user_agent":"Mozilla\/3.0 (compatible; Indy Library)"},"trojanfamily":"Carufax","hostn":"lab14","_provider":"lab","_ts":1381584963,"_geo_env_remote_addr":{"ip":"201.29.117.248","path":"env_remote_addr","country_code":"BR","country_name":"Brazil","region":"Rio de Janeiro","city":"Rio De Janeiro","latitude":-22.8999,"longitude":-43.2333,"asn":7738,"asn_name":"Telemar Norte Leste S.A."}}
{"dtype":"NginxLog","_origin":"banktrojan","env":{"remote_addr":"2.30.182.107"},"trojanfamily":"W32Expiro","data":["POST owyrohikypa.org HTTP\/1.1"],"_provider":"lab","_ts":1381584962,"_geo_env_remote_addr":{"ip":"2.30.182.107","path":"env_remote_addr","country_code":"GB","country_name":"United Kingdom","region":"London, City of","city":"London","latitude":51.5142,"longitude":-0.093,"asn":12576,"asn_name":"Orange Personal Communications Services"}}

Is it possible to upload it directly to splunk storm?

Thanks

Tags (3)
0 Karma

jpass
Contributor

You could create an input script using python or perl that requests the feed. The process might be like this:

  1. perl script would request the url and compare a certain field (date or unique id?) to a stored value called "last_date_checked" or "last_id" which you'd need your script to record somewhere after each run. I use perl module "Config::Simple" which allows me to store a value for this purpose. This value is used to track the returned events so you can determine if the incoming events are new or not. If they're new, print them to screen.

  2. the script simply prints events to the screen

  3. in splunk, add a scripted input and run it at an interval suitable to your needs

http://docs.splunk.com/Documentation/Splunk/6.0/Data/Setupcustominputs

0 Karma

jpass
Contributor

Oh sorry I did not read your request fully. I see you're using Splunk Storm in which case I can't help you.

0 Karma

bizmate
Engager

Is there an API instead to upload files to an index?

0 Karma

okcerto
New Member

Is there anybody out there?!?

0 Karma

okcerto
New Member

Any news about my question?

0 Karma