Getting Data In

upload data directly from a http stream of feeds

okcerto
New Member

Hi there,

I would like to know if there is an option to upload data directly from a http stream of feeds.

Example:

The "http_slash_slash_url_address" returns a something like this:

{"_origin":"banktrojan","env":{"remote_addr":"212.5.158.188"},"trojanfamily":"Rimecud","data":["\b\f\u00c2\u00832C@y\u001b\u00c2\u00b0@"],"hostn":"lab14","_provider":"lab","trojanproto":"udp","_ts":1381584962,"_geo_env_remote_addr":{"ip":"212.5.158.188","path":"env_remote_addr","country_code":"BG","country_name":"Bulgaria","latitude":43,"longitude":25,"asn":8866,"asn_name":"Bulgarian Telecommunication Company Plc."}}
{"_origin":"banktrojan","env":{"remote_addr":"31.174.17.139"},"trojanfamily":"Rimecud","data":["ƒ2#&´\/np"],"hostn":"lab14","_provider":"lab","trojanproto":"udp","_ts":1381584962,"_geo_env_remote_addr":{"ip":"31.174.17.139","path":"env_remote_addr","country_code":"PL","country_name":"Poland","latitude":52,"longitude":20,"asn":39603,"asn_name":"P4 Sp. z o.o."}}
{"_origin":"banktrojan","env":{"remote_addr":"83.150.82.170"},"trojanfamily":"Rimecud","data":["€K"],"hostn":"lab14","_provider":"lab","trojanproto":"udp","_ts":1381584962,"_geo_env_remote_addr":{"ip":"83.150.82.170","path":"env_remote_addr","country_code":"FI","country_name":"Finland","region":"Southern Finland","city":"Helsinki","latitude":60.1756,"longitude":24.9342,"asn":13276,"asn_name":"Nebula Internet international operations AS"}}
{"_origin":"banktrojan","seen":1381584963,"env":{"remote_addr":"182.178.206.99","path_info":"\/ldr.php","request_method":"POST","http_user_agent":"Mozilla\/4.0"},"trojanfamily":"Zeus","hostn":"lab14","_provider":"lab","_ts":1381584963,"_geo_env_remote_addr":{"ip":"182.178.206.99","path":"env_remote_addr","country_code":"PK","country_name":"Pakistan","region":"Punjab","city":"Lahore","latitude":31.5496,"longitude":74.3436,"asn":45595,"asn_name":"Pakistan Telecom Company Limited"}}
{"_origin":"banktrojan","seen":1381584963,"env":{"remote_addr":"201.29.117.248","path_info":"\/print\/eup.html","request_method":"GET","http_user_agent":"Mozilla\/3.0 (compatible; Indy Library)"},"trojanfamily":"Carufax","hostn":"lab14","_provider":"lab","_ts":1381584963,"_geo_env_remote_addr":{"ip":"201.29.117.248","path":"env_remote_addr","country_code":"BR","country_name":"Brazil","region":"Rio de Janeiro","city":"Rio De Janeiro","latitude":-22.8999,"longitude":-43.2333,"asn":7738,"asn_name":"Telemar Norte Leste S.A."}}
{"dtype":"NginxLog","_origin":"banktrojan","env":{"remote_addr":"2.30.182.107"},"trojanfamily":"W32Expiro","data":["POST owyrohikypa.org HTTP\/1.1"],"_provider":"lab","_ts":1381584962,"_geo_env_remote_addr":{"ip":"2.30.182.107","path":"env_remote_addr","country_code":"GB","country_name":"United Kingdom","region":"London, City of","city":"London","latitude":51.5142,"longitude":-0.093,"asn":12576,"asn_name":"Orange Personal Communications Services"}}

Is it possible to upload it directly to splunk storm?

Thanks

Tags (3)
0 Karma

jpass
Contributor

You could create an input script using python or perl that requests the feed. The process might be like this:

  1. perl script would request the url and compare a certain field (date or unique id?) to a stored value called "last_date_checked" or "last_id" which you'd need your script to record somewhere after each run. I use perl module "Config::Simple" which allows me to store a value for this purpose. This value is used to track the returned events so you can determine if the incoming events are new or not. If they're new, print them to screen.

  2. the script simply prints events to the screen

  3. in splunk, add a scripted input and run it at an interval suitable to your needs

http://docs.splunk.com/Documentation/Splunk/6.0/Data/Setupcustominputs

0 Karma

jpass
Contributor

Oh sorry I did not read your request fully. I see you're using Splunk Storm in which case I can't help you.

0 Karma

bizmate
Engager

Is there an API instead to upload files to an index?

0 Karma

okcerto
New Member

Is there anybody out there?!?

0 Karma

okcerto
New Member

Any news about my question?

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud's AI Assistant in Action Series: Auditing Compliance and ...

This is the third post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...