Using Splunk Stream for Netflow- now, ingesting but how to graph?

[keiran_harris]

Hi splunk gurus!

Long weekend here in Australia and i thought id finally get around to ticking something off my wish list: netflow my home network.

So ive got a cisco adsl router thats successfully streaming netflow to my splunk box (verified first with tcpdump). At the splunk side, i started off down one path (“Netflow Analytics” until i realised you had to pay, a lot, for that!)... then some searching in here pointed me to “splunk stream”, which seems robust, is free, now installed, and happily gobbling up my netflow stream! See attached photo.

Which brings me to the fun part (and my question). Where can i find some pre-canned SPL to start plotting my traffic on pretty graphs? The Stream UI doesnt look to be setup for this. I know i could start to write myself but its a relatively complex dataset, and surely this has been done lots before, so i shouldnt have to reinvent the wheel. So if anyone can point me at some SPL (or an app!) that would be great!

Thanks in advance all.
Keiran.

PS- this is the sort of graph I'm hoping to create (from the paid app - https://splunkbase.splunk.com/app/489):

[ygdrassilp]

Hi is there any documentation on how will i configure the netflow on the switch side. And what sourcetype did you use on the udp port 2055? kinda confusing cause im not really familiar with the cisco switch.

Answering this will be a big help for me. Thanksssss!!

[nikhilafedex]

@keiran_harris and @akg2019 can i know how you integrated netflow logs into splunk using stream app, i was going through documentation but its still confusing to me, much appreciate your response on this, Thanks ,@akg2019 and @keiran_harris how did you ingested netflow logs using stream app, i would like to know the process, i went through the splunk documentation but its still little confusing to me, appreciate your response on this

[keiran_harris]

@nikhilafedex - ingesting is the easy part: setup a new (settings -> data inputs -> UDP..... new listener on 2055) + an obviously matching config on your network devices pointing to your splunk instance (on udp 2055).

Then the real work begins - making dashboards fromn the ingested data... i never found time to loo p back on this. Wanted to. Just not managed to.

[astackpole]

Did you do this from a Heavy Forwarder, Search Head, or something else? I followed the documentation as well with the three Netflow applications for a distributed deployment and we're still not seeing any data.

[akg2019]

Hi David,

The new post is yet to be approved. Lets continue working in this thread.

Basically i am looking for network monitoring report via Splunk similar to Manage Engine/Solarwinds/Ipswitch dashboards.

In the report i wanted to calculate metrics such as bitrate (bps) and traffic volume (bytes transferred in MB/GB).
The search query should calculate these metrics for both netflow and sflow data which has the relevant data in different field names.

Sample ingested sflow V5 and netflow V9 data fields are attached.

Can you please help in creating a standard network monitoring report that contains source_IP , dest_IP , Port , Bitrate (bps) , Bytes (MB/GB) etc.. for a given time range.

Thanks,
AKG

[DavidHourani]

so if my understanding is correct, you have the source and dest port in sflow, the bytes_in in netflow and you wish to combine both of them ?

In that case running a search like this should do the trick :

index=yourIndex sourcetype=yoursourcetype | stats values(bytes_in) as bytes_in, values(dest_port) as dest_port values(src_port) as src_port by src_ip, dest_ip | eventstats sum(bytes_in) as volume, avg(bytes_in) as Bps by src_ip, dest_ip

[akg2019]

Hi David,

Thanks for the search query. Still i am confused related to bitrate calculation because avg(bytes_in) will not give bitrate for the time range selected. Below is the specific information i wanted to generate the report.

1. sflow:
   What are the fields that has to be used for calculating bitrate in sflow?
   What is the formula for calculating bitrate from sflow V5 data ?

2. Netflow field : bytes_in
   What is the formula for calculating bitrate from Netflow V9 data ?

Thanks,
AKG

[DavidHourani]

oh yeah... you're right about the bitrate... i think we should be using bytes_in/ (flow_start_time-flow_end_time) to get the exact amount then use an average over that, what do you think ?

[akg2019]

Yes David, For netflow traffic we can use bytes_in field that i am sure. Exact bitrate formula for netflow, i am not sure.

For sflow i could not find any Splunk documentation reference. Any thoughts on the below queries related to sflow ?

1. What are the fields that has to be used for calculating bitrate in sflow?
2. What is the formula for calculating bitrate from sflow V5 data ?

[DavidHourani]

your question is now open 🙂 please upvote my comments here and let's move the discussion there, I will look for any references for bitrate meanwhile ^^

[keiran_harris]

Hi guys, bitrate is simple. Its just: end time minus start time which gives you duration. In this case you derive this from successive events where the flow details match
. Lets say this is 10secs. Then the bits side - if you have a delta of say 1MB (again just minus earlierbit count from later bit count) in that 10 seconds, you just divide by 10 to get the avg bitrate per second... so 100kbps in this example

[DavidHourani]

so something like this : bytes_in/ (flow_start_time-flow_end_time) ?
@keiran_harris

[DavidHourani]

This then makes the search like this :

index=yourIndex sourcetype=yoursourcetype 
|eval bps=bytes_in/(flow_start_time-flow_end_time) 
| stats values(bps) as bps values(bytes_in) as bytes_in, values(dest_port) as dest_port values(src_port) as src_port by src_ip, dest_ip 
| eventstats sum(bytes_in) as volume, avg(bps) as bps by src_ip, dest_ip

[akg2019]

Hi Keiran & David,

sflow does not have bytes_in field. In this case how can we calculate the bitrate?
FYI - I have uploaded the sflow screenshot earlier for reference.

[DavidHourani]

@akg2019, that's why I'm using the values command. This regroups all the values regardless of whether they are from sflow or netflowdata. This means we can regroup information from both types of flows and then build the search.

[akg2019]

Hi David,

Sflow and netflow are two seperate events with unique field names. For example bytes_in is not present in sflow event. In this case can you please explain how bitrate will be calculated based on your above search query ?

[DavidHourani]

sure.

First part index=yourIndex sourcetype=yoursourcetype should call data from both netflow and sflow.

Second part |eval bps=bytes_in/(flow_start_time-flow_end_time) adds a new field to the Netflow data.

Third part | stats values(bps) as bps values(bytes_in) as bytes_in, values(dest_port) as dest_port values(src_port) as src_port by src_ip, dest_ip will find values of bps, values of bytes_in values of dest_port and values of src for all combinations of src_ip and dest_ip regardless of where they are (netflow and sflow)

Last part simply sums up the volume and gets the average bps 😄

[akg2019]

Hi Keiran,

If you see my sflow V5 screen shot the below fields can be found.

1. src_ip (I have erased the IP details in the screenshot for security reasons)
2. src_port
3. dest_ip
4. dest_port (I have erased the IP details in the screenshot for security reasons)

[DavidHourani]

@keiran_harris,

Thanks for the explaining this ! using seqnumber is a good idea for getting the volume of a session sadly netflowdata doesn't report src port and dest port, it just shows zeros, which brings us back to using sflow data alone and bytes_in isn't there + seqnumber doesn't match with netflow's seq, I guess one is a seqnumber for the session and the other for the entire communication ?

@akg2019 does the search I sent above make sense to you ?

[akg2019]

Hi David,

netflow data reports dest_port and src_port.
In my sample screenshot these values are 0. However in my splunk server the netflow events has values.
The netflow event screenshot shared by Keiran at the start of this post also has values for dest_port and src_port.

The search query you shared did not work for sflow events.