upload data directly from a http stream of feeds

okcerto — Mon, 28 Sep 2020 14:57:33 GMT

Hi there,

I would like to know if there is an option to upload data directly from a http stream of feeds.

Example:

The "http_slash_slash_url_address" returns a something like this:

{"_origin":"banktrojan","env":{"remote_addr":"212.5.158.188"},"trojanfamily":"Rimecud","data":["\b\f\u00c2\u00832C@y\u001b\u00c2\u00b0@"],"hostn":"lab14","_provider":"lab","trojanproto":"udp","_ts":1381584962,"_geo_env_remote_addr":{"ip":"212.5.158.188","path":"env_remote_addr","country_code":"BG","country_name":"Bulgaria","latitude":43,"longitude":25,"asn":8866,"asn_name":"Bulgarian Telecommunication Company Plc."}}
{"_origin":"banktrojan","env":{"remote_addr":"31.174.17.139"},"trojanfamily":"Rimecud","data":["ƒ2#&´\/np"],"hostn":"lab14","_provider":"lab","trojanproto":"udp","_ts":1381584962,"_geo_env_remote_addr":{"ip":"31.174.17.139","path":"env_remote_addr","country_code":"PL","country_name":"Poland","latitude":52,"longitude":20,"asn":39603,"asn_name":"P4 Sp. z o.o."}}
{"_origin":"banktrojan","env":{"remote_addr":"83.150.82.170"},"trojanfamily":"Rimecud","data":["€K"],"hostn":"lab14","_provider":"lab","trojanproto":"udp","_ts":1381584962,"_geo_env_remote_addr":{"ip":"83.150.82.170","path":"env_remote_addr","country_code":"FI","country_name":"Finland","region":"Southern Finland","city":"Helsinki","latitude":60.1756,"longitude":24.9342,"asn":13276,"asn_name":"Nebula Internet international operations AS"}}
{"_origin":"banktrojan","seen":1381584963,"env":{"remote_addr":"182.178.206.99","path_info":"\/ldr.php","request_method":"POST","http_user_agent":"Mozilla\/4.0"},"trojanfamily":"Zeus","hostn":"lab14","_provider":"lab","_ts":1381584963,"_geo_env_remote_addr":{"ip":"182.178.206.99","path":"env_remote_addr","country_code":"PK","country_name":"Pakistan","region":"Punjab","city":"Lahore","latitude":31.5496,"longitude":74.3436,"asn":45595,"asn_name":"Pakistan Telecom Company Limited"}}
{"_origin":"banktrojan","seen":1381584963,"env":{"remote_addr":"201.29.117.248","path_info":"\/print\/eup.html","request_method":"GET","http_user_agent":"Mozilla\/3.0 (compatible; Indy Library)"},"trojanfamily":"Carufax","hostn":"lab14","_provider":"lab","_ts":1381584963,"_geo_env_remote_addr":{"ip":"201.29.117.248","path":"env_remote_addr","country_code":"BR","country_name":"Brazil","region":"Rio de Janeiro","city":"Rio De Janeiro","latitude":-22.8999,"longitude":-43.2333,"asn":7738,"asn_name":"Telemar Norte Leste S.A."}}
{"dtype":"NginxLog","_origin":"banktrojan","env":{"remote_addr":"2.30.182.107"},"trojanfamily":"W32Expiro","data":["POST owyrohikypa.org HTTP\/1.1"],"_provider":"lab","_ts":1381584962,"_geo_env_remote_addr":{"ip":"2.30.182.107","path":"env_remote_addr","country_code":"GB","country_name":"United Kingdom","region":"London, City of","city":"London","latitude":51.5142,"longitude":-0.093,"asn":12576,"asn_name":"Orange Personal Communications Services"}}

Is it possible to upload it directly to splunk storm?

Thanks

Re: upload data directly from a http stream of feeds

okcerto — Mon, 14 Oct 2013 16:57:07 GMT

Any news about my question?

Re: upload data directly from a http stream of feeds

okcerto — Tue, 15 Oct 2013 14:34:02 GMT

Is there anybody out there?!?

Re: upload data directly from a http stream of feeds

jpass — Mon, 28 Sep 2020 14:58:38 GMT

You could create an input script using python or perl that requests the feed. The process might be like this:

perl script would request the url and compare a certain field (date or unique id?) to a stored value called "last_date_checked" or "last_id" which you'd need your script to record somewhere after each run. I use perl module "Config::Simple" which allows me to store a value for this purpose. This value is used to track the returned events so you can determine if the incoming events are new or not. If they're new, print them to screen.
the script simply prints events to the screen
in splunk, add a scripted input and run it at an interval suitable to your needs

http://docs.splunk.com/Documentation/Splunk/6.0/Data/Setupcustominputs

Re: upload data directly from a http stream of feeds

jpass — Tue, 15 Oct 2013 15:14:47 GMT

Oh sorry I did not read your request fully. I see you're using Splunk Storm in which case I can't help you.

Re: upload data directly from a http stream of feeds

bizmate — Wed, 02 Mar 2016 22:18:42 GMT

Is there an API instead to upload files to an index?

topic Re: upload data directly from a http stream of feeds in Getting Data In

upload data directly from a http stream of feeds

Re: upload data directly from a http stream of feeds

Re: upload data directly from a http stream of feeds

Re: upload data directly from a http stream of feeds

Re: upload data directly from a http stream of feeds

Re: upload data directly from a http stream of feeds