All Apps and Add-ons

Using Splunk Stream for Netflow- now, ingesting but how to graph?

keiran_harris
Path Finder

Hi splunk gurus!

Long weekend here in Australia and i thought id finally get around to ticking something off my wish list: netflow my home network.

So ive got a cisco adsl router thats successfully streaming netflow to my splunk box (verified first with tcpdump). At the splunk side, i started off down one path (“Netflow Analytics” until i realised you had to pay, a lot, for that!)... then some searching in here pointed me to “splunk stream”, which seems robust, is free, now installed, and happily gobbling up my netflow stream! See attached photo.

alt text

Which brings me to the fun part (and my question). Where can i find some pre-canned SPL to start plotting my traffic on pretty graphs? The Stream UI doesnt look to be setup for this. I know i could start to write myself but its a relatively complex dataset, and surely this has been done lots before, so i shouldnt have to reinvent the wheel. So if anyone can point me at some SPL (or an app!) that would be great!

Thanks in advance all.
Keiran.

PS- this is the sort of graph I'm hoping to create (from the paid app - https://splunkbase.splunk.com/app/489):
alt text

ygdrassilp
Explorer

Hi is there any documentation on how will i configure the netflow on the switch side. And what sourcetype did you use on the udp port 2055? kinda confusing cause im not really familiar with the cisco switch.

Answering this will be a big help for me. Thanksssss!!

0 Karma

nikhilafedex
New Member

@keiran_harris and @akg2019 can i know how you integrated netflow logs into splunk using stream app, i was going through documentation but its still confusing to me, much appreciate your response on this, Thanks ,@akg2019 and @keiran_harris how did you ingested netflow logs using stream app, i would like to know the process, i went through the splunk documentation but its still little confusing to me, appreciate your response on this

0 Karma

keiran_harris
Path Finder

@nikhilafedex - ingesting is the easy part: setup a new (settings -> data inputs -> UDP..... new listener on 2055) + an obviously matching config on your network devices pointing to your splunk instance (on udp 2055).

Then the real work begins - making dashboards fromn the ingested data... i never found time to loo p back on this. Wanted to. Just not managed to.

0 Karma

astackpole
Path Finder

Did you do this from a Heavy Forwarder, Search Head, or something else? I followed the documentation as well with the three Netflow applications for a distributed deployment and we're still not seeing any data.

0 Karma

akg2019
Explorer

Hi David,

The new post is yet to be approved. Lets continue working in this thread.

Basically i am looking for network monitoring report via Splunk similar to Manage Engine/Solarwinds/Ipswitch dashboards.

In the report i wanted to calculate metrics such as bitrate (bps) and traffic volume (bytes transferred in MB/GB).
The search query should calculate these metrics for both netflow and sflow data which has the relevant data in different field names.

Sample ingested sflow V5 and netflow V9 data fields are attached.

alt text

alt text

Can you please help in creating a standard network monitoring report that contains source_IP , dest_IP , Port , Bitrate (bps) , Bytes (MB/GB) etc.. for a given time range.

Thanks,
AKG

0 Karma

DavidHourani
Super Champion

so if my understanding is correct, you have the source and dest port in sflow, the bytes_in in netflow and you wish to combine both of them ?

In that case running a search like this should do the trick :

index=yourIndex sourcetype=yoursourcetype | stats values(bytes_in) as bytes_in, values(dest_port) as dest_port values(src_port) as src_port by src_ip, dest_ip | eventstats sum(bytes_in) as volume, avg(bytes_in) as Bps by src_ip, dest_ip
0 Karma

akg2019
Explorer

Hi David,

Thanks for the search query. Still i am confused related to bitrate calculation because avg(bytes_in) will not give bitrate for the time range selected. Below is the specific information i wanted to generate the report.

  1. sflow:
    What are the fields that has to be used for calculating bitrate in sflow?
    What is the formula for calculating bitrate from sflow V5 data ?

  2. Netflow field : bytes_in
    What is the formula for calculating bitrate from Netflow V9 data ?

Thanks,
AKG

0 Karma

DavidHourani
Super Champion

oh yeah... you're right about the bitrate... i think we should be using bytes_in/ (flow_start_time-flow_end_time) to get the exact amount then use an average over that, what do you think ?

0 Karma

akg2019
Explorer

Yes David, For netflow traffic we can use bytes_in field that i am sure. Exact bitrate formula for netflow, i am not sure.

For sflow i could not find any Splunk documentation reference. Any thoughts on the below queries related to sflow ?

  1. What are the fields that has to be used for calculating bitrate in sflow?
  2. What is the formula for calculating bitrate from sflow V5 data ?
0 Karma

DavidHourani
Super Champion

your question is now open 🙂 please upvote my comments here and let's move the discussion there, I will look for any references for bitrate meanwhile ^^

0 Karma

keiran_harris
Path Finder

Hi guys, bitrate is simple. Its just: end time minus start time which gives you duration. In this case you derive this from successive events where the flow details match
. Lets say this is 10secs. Then the bits side - if you have a delta of say 1MB (again just minus earlierbit count from later bit count) in that 10 seconds, you just divide by 10 to get the avg bitrate per second... so 100kbps in this example

0 Karma

DavidHourani
Super Champion

so something like this : bytes_in/ (flow_start_time-flow_end_time) ?
@keiran_harris

0 Karma

DavidHourani
Super Champion

This then makes the search like this :

index=yourIndex sourcetype=yoursourcetype 
|eval bps=bytes_in/(flow_start_time-flow_end_time) 
| stats values(bps) as bps values(bytes_in) as bytes_in, values(dest_port) as dest_port values(src_port) as src_port by src_ip, dest_ip 
| eventstats sum(bytes_in) as volume, avg(bps) as bps by src_ip, dest_ip
0 Karma

akg2019
Explorer

Hi Keiran & David,

sflow does not have bytes_in field. In this case how can we calculate the bitrate?
FYI - I have uploaded the sflow screenshot earlier for reference.

0 Karma

DavidHourani
Super Champion

@akg2019, that's why I'm using the values command. This regroups all the values regardless of whether they are from sflow or netflowdata. This means we can regroup information from both types of flows and then build the search.

0 Karma

akg2019
Explorer

Hi David,

Sflow and netflow are two seperate events with unique field names. For example bytes_in is not present in sflow event. In this case can you please explain how bitrate will be calculated based on your above search query ?

0 Karma

DavidHourani
Super Champion

sure.

First part index=yourIndex sourcetype=yoursourcetype should call data from both netflow and sflow.

Second part |eval bps=bytes_in/(flow_start_time-flow_end_time) adds a new field to the Netflow data.

Third part | stats values(bps) as bps values(bytes_in) as bytes_in, values(dest_port) as dest_port values(src_port) as src_port by src_ip, dest_ip will find values of bps, values of bytes_in values of dest_port and values of src for all combinations of src_ip and dest_ip regardless of where they are (netflow and sflow)

Last part simply sums up the volume and gets the average bps 😄

0 Karma

akg2019
Explorer

Hi Keiran,

If you see my sflow V5 screen shot the below fields can be found.

  1. src_ip (I have erased the IP details in the screenshot for security reasons)
  2. src_port
  3. dest_ip
  4. dest_port (I have erased the IP details in the screenshot for security reasons)
0 Karma

DavidHourani
Super Champion

@keiran_harris,

Thanks for the explaining this ! using seqnumber is a good idea for getting the volume of a session sadly netflowdata doesn't report src port and dest port, it just shows zeros, which brings us back to using sflow data alone and bytes_in isn't there + seqnumber doesn't match with netflow's seq, I guess one is a seqnumber for the session and the other for the entire communication ?

@akg2019 does the search I sent above make sense to you ?

0 Karma

akg2019
Explorer

Hi David,

netflow data reports dest_port and src_port.
In my sample screenshot these values are 0. However in my splunk server the netflow events has values.
The netflow event screenshot shared by Keiran at the start of this post also has values for dest_port and src_port.

The search query you shared did not work for sflow events.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...