Getting Data In
Highlighted

upload data directly from a http stream of feeds

New Member

Hi there,

I would like to know if there is an option to upload data directly from a http stream of feeds.

Example:

The "httpslashslashurladdress" returns a something like this:

{"origin":"banktrojan","env":{"remoteaddr":"212.5.158.188"},"trojanfamily":"Rimecud","data":["\b\f\u00c2\u00832C@y\u001b\u00c2\u00b0@"],"hostn":"lab14","provider":"lab","trojanproto":"udp","ts":1381584962,"geoenvremoteaddr":{"ip":"212.5.158.188","path":"envremoteaddr","countrycode":"BG","countryname":"Bulgaria","latitude":43,"longitude":25,"asn":8866,"asnname":"Bulgarian Telecommunication Company Plc."}}
{"
origin":"banktrojan","env":{"remoteaddr":"31.174.17.139"},"trojanfamily":"Rimecud","data":["ƒ2#&´\/np"],"hostn":"lab14","provider":"lab","trojanproto":"udp","ts":1381584962,"geoenvremoteaddr":{"ip":"31.174.17.139","path":"envremoteaddr","countrycode":"PL","countryname":"Poland","latitude":52,"longitude":20,"asn":39603,"asnname":"P4 Sp. z o.o."}}
{"origin":"banktrojan","env":{"remoteaddr":"83.150.82.170"},"trojanfamily":"Rimecud","data":["€K"],"hostn":"lab14","provider":"lab","trojanproto":"udp","ts":1381584962,"geoenvremoteaddr":{"ip":"83.150.82.170","path":"envremoteaddr","countrycode":"FI","countryname":"Finland","region":"Southern Finland","city":"Helsinki","latitude":60.1756,"longitude":24.9342,"asn":13276,"asnname":"Nebula Internet international operations AS"}}
{"
origin":"banktrojan","seen":1381584963,"env":{"remoteaddr":"182.178.206.99","pathinfo":"\/ldr.php","requestmethod":"POST","httpuseragent":"Mozilla\/4.0"},"trojanfamily":"Zeus","hostn":"lab14","provider":"lab","ts":1381584963,"geoenvremoteaddr":{"ip":"182.178.206.99","path":"envremoteaddr","countrycode":"PK","countryname":"Pakistan","region":"Punjab","city":"Lahore","latitude":31.5496,"longitude":74.3436,"asn":45595,"asnname":"Pakistan Telecom Company Limited"}}
{"origin":"banktrojan","seen":1381584963,"env":{"remoteaddr":"201.29.117.248","pathinfo":"\/print\/eup.html","requestmethod":"GET","httpuseragent":"Mozilla\/3.0 (compatible; Indy Library)"},"trojanfamily":"Carufax","hostn":"lab14","provider":"lab","ts":1381584963,"geoenvremoteaddr":{"ip":"201.29.117.248","path":"envremoteaddr","countrycode":"BR","countryname":"Brazil","region":"Rio de Janeiro","city":"Rio De Janeiro","latitude":-22.8999,"longitude":-43.2333,"asn":7738,"asnname":"Telemar Norte Leste S.A."}}
{"dtype":"NginxLog","
origin":"banktrojan","env":{"remoteaddr":"2.30.182.107"},"trojanfamily":"W32Expiro","data":["POST owyrohikypa.org HTTP\/1.1"],"provider":"lab","ts":1381584962,"geoenvremoteaddr":{"ip":"2.30.182.107","path":"envremoteaddr","countrycode":"GB","countryname":"United Kingdom","region":"London, City of","city":"London","latitude":51.5142,"longitude":-0.093,"asn":12576,"asnname":"Orange Personal Communications Services"}}

Is it possible to upload it directly to splunk storm?

Thanks

Tags (3)
0 Karma
Highlighted

Re: upload data directly from a http stream of feeds

New Member

Any news about my question?

0 Karma
Highlighted

Re: upload data directly from a http stream of feeds

New Member

Is there anybody out there?!?

0 Karma
Highlighted

Re: upload data directly from a http stream of feeds

Contributor

You could create an input script using python or perl that requests the feed. The process might be like this:

  1. perl script would request the url and compare a certain field (date or unique id?) to a stored value called "lastdatechecked" or "last_id" which you'd need your script to record somewhere after each run. I use perl module "Config::Simple" which allows me to store a value for this purpose. This value is used to track the returned events so you can determine if the incoming events are new or not. If they're new, print them to screen.

  2. the script simply prints events to the screen

  3. in splunk, add a scripted input and run it at an interval suitable to your needs

http://docs.splunk.com/Documentation/Splunk/6.0/Data/Setupcustominputs

0 Karma
Highlighted

Re: upload data directly from a http stream of feeds

Contributor

Oh sorry I did not read your request fully. I see you're using Splunk Storm in which case I can't help you.

0 Karma
Highlighted

Re: upload data directly from a http stream of feeds

Engager

Is there an API instead to upload files to an index?

0 Karma