All Apps and Add-ons

Using Splunk Stream for Netflow- now, ingesting but how to graph?

keiran_harris
Path Finder

Hi splunk gurus!

Long weekend here in Australia and i thought id finally get around to ticking something off my wish list: netflow my home network.

So ive got a cisco adsl router thats successfully streaming netflow to my splunk box (verified first with tcpdump). At the splunk side, i started off down one path (“Netflow Analytics” until i realised you had to pay, a lot, for that!)... then some searching in here pointed me to “splunk stream”, which seems robust, is free, now installed, and happily gobbling up my netflow stream! See attached photo.

alt text

Which brings me to the fun part (and my question). Where can i find some pre-canned SPL to start plotting my traffic on pretty graphs? The Stream UI doesnt look to be setup for this. I know i could start to write myself but its a relatively complex dataset, and surely this has been done lots before, so i shouldnt have to reinvent the wheel. So if anyone can point me at some SPL (or an app!) that would be great!

Thanks in advance all.
Keiran.

PS- this is the sort of graph I'm hoping to create (from the paid app - https://splunkbase.splunk.com/app/489):
alt text

DavidHourani
Super Champion

ohh! okay, good, well the sflow events are taken into consideration for sure, but since they don't have a bytes_in value then yeah you won't get any Bps/volume values. We can try with the seqnumber for sflow...comparing with previous events and then using the delta and duration to estimate the bandwidth. What do you think ?

0 Karma

keiran_harris
Path Finder

sounds good to me! we may have to tweak the logic when you get a graph going based on some of my assumptions (for instance ive been thinking in the netflow bytes_in field is that bytes since last update, or cumulate to that point - like the TCP seqnumber). If you want some same live data, i could look into exporting some for you.

0 Karma

keiran_harris
Path Finder

Hi @akg2019 how did you get on with this? I haven’t had a chance to circle back yet. Keen to understand how you got on.

0 Karma

keiran_harris
Path Finder

Hi @akg2019 - i think your problem / question around sFlow is a more fundamental one. Im a long time network engineer so i might be able to shed light on the different datasets.

Netflow is an accurate measure of traffic (bits)- actually it was/still is used for many billing systems for instance to track what customer consumed what data.

sFlow on the other hand is not. The clue is in the name - its sampled. Its evolution was driven by faster network kit, where netflow (tracking bit count on EVERY session would flatten the CPU of the router). So in sFlow, every so often, might be one packet in 10,000, the sflow process will wake up, peek in at that packet transiting the box at that time and report on that packet, then the process will disappear... and reappear again to check in at the next sampled interval (another 10,000 packets). The logic is that the sampling will be able to roughly report on the transiting traffic.... as big / long-running / high-bandwidth sessions are more likely to be hit upon by the sampling.

In sFlow, I dont believe the concept of sessions (src_ip + src_port + dst_ip + dst_port) are tracked, which is necessary for the router to keep track of incrementing bit count like it does in netflow... so the fact that bytes_in field is not present in sFlow makes perfect sense to me.

In saying that i know solarwinds etc have developed an interpretation of this data. Graphing it doesnt make sense to me given the above. What does it look like? Ahh just checked, seems its tabular reporting (see https://www.solarwinds.com/topics/sflow-collector) which does make more sense. Those tables do have byte count though hmmm... how wuld they get that.... (checking your sflow packet sample now)....

OK there is a seqnumber field - and in TCP at least thats an incremtal count of the bytes transferred so far, and is included in EVERY packet as a running total. So i guess thats how they do it, and thats what you likely need to report on. But that only exists in TCP (UDP for instance does not have this).

hope this helps.

As for me, i will likely have time to mess around with this stuff on the wekeend.
Ever grateful for your assistance here @DavidHourani

thanks
Keiran.

0 Karma

keiran_harris
Path Finder

Hi all, i havent had time to look at this further. My splunk is still ingesting loads of netflow, but i havent started dev on the SPL. Seems lots of people looking for this. @DavidHourani has specifically asked for a new question to be asked on a new post, not quite sure why, its still the same dev problem we need solved, but regardless happy to follow the new thread, just pls link us in here @akg2019 so we know where to follow. Thanks guys!

0 Karma

keiran_harris
Path Finder

Hi @DavidHourani - really appreciate your assistance here.... attached is a screenshot of some sample data thats as good as any other. Let me know if you need an actual export.

alt text

Basically (if you didnt know about netflow) the router sends periodical "flow records" back to a reciever - in this case splunk (FYI - each data packet can contain many flow records, and splunk pulls them out as an event per record).... so its a snapshot into what the routers session table is at that moment, inclusive of byte count for those transiting sessions. So if you have a long running TCP session to a DB server for instance, at minute one, it will have a byte count (bytes_in/out) of say 100.... check back 1 minute later, it migth have a byte count of say 1000, indicating 900 more bytes in that last minute.

I think the search logic needs to
- group like flows by TCP/UDP sessions which is (src_ip + dst_ip + src_port + dest_port).....
- graphing bytes over time.
- And then grabbing only say the top 10 flows by byte count.
Check the original post for the kind of flow data visualisation over time we are hoping for.

0 Karma

keiran_harris
Path Finder

^ and im not sure if you missed this comment @DavidHourani
Does my logic make sense, and if so do your search query's cover it?
i will test in thew weekend.

0 Karma

akg2019
Explorer

Hi David and Keiran,

I have created a new post. Please follow the below link.

https://answers.splunk.com/answers/747044/how-to-create-network-monitoring-report-for-netflo.html?mi...

Subject : How to create network monitoring report for netflow and sflow data ?

Thanks,
AKG

DavidHourani
Super Champion

link is not working for me XD

0 Karma

akg2019
Explorer

Hi David,

The new post status is "This post is currently awaiting moderation. If you believe this to be in error, contact a system administrator."

Not sure when it will get approved.

0 Karma

DavidHourani
Super Champion

ouch... okay, let me know when it's up, and if you want go ahead and share some sample (anonymized) logs here so we can work with it.

0 Karma

tobiasgoevert
Engager

Hi Together,

same situation for me!
i ingested netflow from our cisco-routers to splunk via Splunk app for stream.
Now i want to visualize it.

@keiran_harris do you have some results jet?

Regards, Tobias,Hi together,

same situation for me!
We also ingested netflow from our cisco-router and want to visualize it now.
@keiran_harris do you have some results jet?

Regards, Tobias

0 Karma

akg2019
Explorer

Hi,

I have ingested netflow and sflow wire data from our Juniper switches. But there is no visualization app with inbuilt/default dashboards. Can someone help with SPL queries or apps that can visualize the data similar to Manage Engine/Solarwinds dashboards?

Thanks,
AKG

0 Karma

DavidHourani
Super Champion

sure, what are you trying to build ?

0 Karma

akg2019
Explorer

Hi David,

I am trying to create custom dashboard report that lists the top N source to destination conversation by bit rate (bps) and traffic volume (Total MB/GB).

Post this i wanted to include other fields like port and Interface ID's as well.

Thanks,
AKG

0 Karma

DavidHourani
Super Champion

@akg2019, you're looking for something like this :

index=whereYourDataIs sourcetype=yourSourcetype | stats avg(bps) as bitRate, sum(bps) as volume by src dest
0 Karma

akg2019
Explorer

Hi David,
Thanks for the search query. However bps is not captured directly. For example in sflow data there is no field such as bps. It has to be calculated manually. Same applies to netflow as well.

I am looking for the search queries that calculates bitrate (bps) and traffic volume (bytes transferred in MB/GB). The search query should calculate these metrics for both netflow and sflow data which has the relevant data in different field names.

Basically i am looking for network monitoring report via Splunk. Any help on this is highly appreciated.

0 Karma

DavidHourani
Super Champion

Can you make it into a new question please and include a sample event line ? We can work from there

0 Karma

akg2019
Explorer

Sure David

0 Karma

keiran_harris
Path Finder

Giving this a nudge so it bubbles up again for some viewers who can help!

0 Karma

xiongwei002
New Member

Agree, top up, I need it too.

0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...