All Apps and Add-ons

Using Splunk Stream for Netflow- now, ingesting but how to graph?

Path Finder

Hi splunk gurus!

Long weekend here in Australia and i thought id finally get around to ticking something off my wish list: netflow my home network.

So ive got a cisco adsl router thats successfully streaming netflow to my splunk box (verified first with tcpdump). At the splunk side, i started off down one path (“Netflow Analytics” until i realised you had to pay, a lot, for that!)... then some searching in here pointed me to “splunk stream”, which seems robust, is free, now installed, and happily gobbling up my netflow stream! See attached photo.

alt text

Which brings me to the fun part (and my question). Where can i find some pre-canned SPL to start plotting my traffic on pretty graphs? The Stream UI doesnt look to be setup for this. I know i could start to write myself but its a relatively complex dataset, and surely this has been done lots before, so i shouldnt have to reinvent the wheel. So if anyone can point me at some SPL (or an app!) that would be great!

Thanks in advance all.
Keiran.

PS- this is the sort of graph I'm hoping to create (from the paid app - https://splunkbase.splunk.com/app/489):
alt text

Explorer

Hi is there any documentation on how will i configure the netflow on the switch side. And what sourcetype did you use on the udp port 2055? kinda confusing cause im not really familiar with the cisco switch.

Answering this will be a big help for me. Thanksssss!!

0 Karma

New Member

@keiran_harris and @akg2019 can i know how you integrated netflow logs into splunk using stream app, i was going through documentation but its still confusing to me, much appreciate your response on this, Thanks ,@akg2019 and @keiran_harris how did you ingested netflow logs using stream app, i would like to know the process, i went through the splunk documentation but its still little confusing to me, appreciate your response on this

0 Karma

Path Finder

@nikhilafedex - ingesting is the easy part: setup a new (settings -> data inputs -> UDP..... new listener on 2055) + an obviously matching config on your network devices pointing to your splunk instance (on udp 2055).

Then the real work begins - making dashboards fromn the ingested data... i never found time to loo p back on this. Wanted to. Just not managed to.

0 Karma

Explorer

Hi David,

The new post is yet to be approved. Lets continue working in this thread.

Basically i am looking for network monitoring report via Splunk similar to Manage Engine/Solarwinds/Ipswitch dashboards.

In the report i wanted to calculate metrics such as bitrate (bps) and traffic volume (bytes transferred in MB/GB).
The search query should calculate these metrics for both netflow and sflow data which has the relevant data in different field names.

Sample ingested sflow V5 and netflow V9 data fields are attached.

alt text

alt text

Can you please help in creating a standard network monitoring report that contains source_IP , dest_IP , Port , Bitrate (bps) , Bytes (MB/GB) etc.. for a given time range.

Thanks,
AKG

0 Karma

Super Champion

so if my understanding is correct, you have the source and dest port in sflow, the bytes_in in netflow and you wish to combine both of them ?

In that case running a search like this should do the trick :

index=yourIndex sourcetype=yoursourcetype | stats values(bytes_in) as bytes_in, values(dest_port) as dest_port values(src_port) as src_port by src_ip, dest_ip | eventstats sum(bytes_in) as volume, avg(bytes_in) as Bps by src_ip, dest_ip
0 Karma

Explorer

Hi David,

Thanks for the search query. Still i am confused related to bitrate calculation because avg(bytes_in) will not give bitrate for the time range selected. Below is the specific information i wanted to generate the report.

  1. sflow:
    What are the fields that has to be used for calculating bitrate in sflow?
    What is the formula for calculating bitrate from sflow V5 data ?

  2. Netflow field : bytes_in
    What is the formula for calculating bitrate from Netflow V9 data ?

Thanks,
AKG

0 Karma

Super Champion

oh yeah... you're right about the bitrate... i think we should be using bytes_in/ (flow_start_time-flow_end_time) to get the exact amount then use an average over that, what do you think ?

0 Karma

Explorer

Yes David, For netflow traffic we can use bytes_in field that i am sure. Exact bitrate formula for netflow, i am not sure.

For sflow i could not find any Splunk documentation reference. Any thoughts on the below queries related to sflow ?

  1. What are the fields that has to be used for calculating bitrate in sflow?
  2. What is the formula for calculating bitrate from sflow V5 data ?
0 Karma

Super Champion

your question is now open 🙂 please upvote my comments here and let's move the discussion there, I will look for any references for bitrate meanwhile ^^

0 Karma

Path Finder

Hi guys, bitrate is simple. Its just: end time minus start time which gives you duration. In this case you derive this from successive events where the flow details match
. Lets say this is 10secs. Then the bits side - if you have a delta of say 1MB (again just minus earlierbit count from later bit count) in that 10 seconds, you just divide by 10 to get the avg bitrate per second... so 100kbps in this example

0 Karma

Super Champion

so something like this : bytes_in/ (flow_start_time-flow_end_time) ?
@keiran_harris

0 Karma

Super Champion

This then makes the search like this :

index=yourIndex sourcetype=yoursourcetype 
|eval bps=bytes_in/(flow_start_time-flow_end_time) 
| stats values(bps) as bps values(bytes_in) as bytes_in, values(dest_port) as dest_port values(src_port) as src_port by src_ip, dest_ip 
| eventstats sum(bytes_in) as volume, avg(bps) as bps by src_ip, dest_ip
0 Karma

Explorer

Hi Keiran & David,

sflow does not have bytes_in field. In this case how can we calculate the bitrate?
FYI - I have uploaded the sflow screenshot earlier for reference.

0 Karma

Super Champion

@akg2019, that's why I'm using the values command. This regroups all the values regardless of whether they are from sflow or netflowdata. This means we can regroup information from both types of flows and then build the search.

0 Karma

Explorer

Hi David,

Sflow and netflow are two seperate events with unique field names. For example bytes_in is not present in sflow event. In this case can you please explain how bitrate will be calculated based on your above search query ?

0 Karma

Super Champion

sure.

First part index=yourIndex sourcetype=yoursourcetype should call data from both netflow and sflow.

Second part |eval bps=bytes_in/(flow_start_time-flow_end_time) adds a new field to the Netflow data.

Third part | stats values(bps) as bps values(bytes_in) as bytes_in, values(dest_port) as dest_port values(src_port) as src_port by src_ip, dest_ip will find values of bps, values of bytes_in values of dest_port and values of src for all combinations of src_ip and dest_ip regardless of where they are (netflow and sflow)

Last part simply sums up the volume and gets the average bps 😄

0 Karma

Explorer

Hi Keiran,

If you see my sflow V5 screen shot the below fields can be found.

  1. src_ip (I have erased the IP details in the screenshot for security reasons)
  2. src_port
  3. dest_ip
  4. dest_port (I have erased the IP details in the screenshot for security reasons)
0 Karma

Super Champion

@keiran_harris,

Thanks for the explaining this ! using seqnumber is a good idea for getting the volume of a session sadly netflowdata doesn't report src port and dest port, it just shows zeros, which brings us back to using sflow data alone and bytes_in isn't there + seqnumber doesn't match with netflow's seq, I guess one is a seqnumber for the session and the other for the entire communication ?

@akg2019 does the search I sent above make sense to you ?

0 Karma

Explorer

Hi David,

netflow data reports dest_port and src_port.
In my sample screenshot these values are 0. However in my splunk server the netflow events has values.
The netflow event screenshot shared by Keiran at the start of this post also has values for dest_port and src_port.

The search query you shared did not work for sflow events.

0 Karma

Super Champion

ohh! okay, good, well the sflow events are taken into consideration for sure, but since they don't have a bytes_in value then yeah you won't get any Bps/volume values. We can try with the seqnumber for sflow...comparing with previous events and then using the delta and duration to estimate the bandwidth. What do you think ?

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!