Hi splunk gurus!
Long weekend here in Australia and i thought id finally get around to ticking something off my wish list: netflow my home network.
So ive got a cisco adsl router thats successfully streaming netflow to my splunk box (verified first with tcpdump). At the splunk side, i started off down one path (“Netflow Analytics” until i realised you had to pay, a lot, for that!)... then some searching in here pointed me to “splunk stream”, which seems robust, is free, now installed, and happily gobbling up my netflow stream! See attached photo.
Which brings me to the fun part (and my question). Where can i find some pre-canned SPL to start plotting my traffic on pretty graphs? The Stream UI doesnt look to be setup for this. I know i could start to write myself but its a relatively complex dataset, and surely this has been done lots before, so i shouldnt have to reinvent the wheel. So if anyone can point me at some SPL (or an app!) that would be great!
Thanks in advance all.
PS- this is the sort of graph I'm hoping to create (from the paid app - https://splunkbase.splunk.com/app/489):
sounds good to me! we may have to tweak the logic when you get a graph going based on some of my assumptions (for instance ive been thinking in the netflow bytes_in field is that bytes since last update, or cumulate to that point - like the TCP seqnumber). If you want some same live data, i could look into exporting some for you.
Hi @akg2019 - i think your problem / question around sFlow is a more fundamental one. Im a long time network engineer so i might be able to shed light on the different datasets.
Netflow is an accurate measure of traffic (bits)- actually it was/still is used for many billing systems for instance to track what customer consumed what data.
sFlow on the other hand is not. The clue is in the name - its sampled. Its evolution was driven by faster network kit, where netflow (tracking bit count on EVERY session would flatten the CPU of the router). So in sFlow, every so often, might be one packet in 10,000, the sflow process will wake up, peek in at that packet transiting the box at that time and report on that packet, then the process will disappear... and reappear again to check in at the next sampled interval (another 10,000 packets). The logic is that the sampling will be able to roughly report on the transiting traffic.... as big / long-running / high-bandwidth sessions are more likely to be hit upon by the sampling.
In sFlow, I dont believe the concept of sessions (src_ip + src_port + dst_ip + dst_port) are tracked, which is necessary for the router to keep track of incrementing bit count like it does in netflow... so the fact that bytes_in field is not present in sFlow makes perfect sense to me.
In saying that i know solarwinds etc have developed an interpretation of this data. Graphing it doesnt make sense to me given the above. What does it look like? Ahh just checked, seems its tabular reporting (see https://www.solarwinds.com/topics/sflow-collector) which does make more sense. Those tables do have byte count though hmmm... how wuld they get that.... (checking your sflow packet sample now)....
OK there is a seqnumber field - and in TCP at least thats an incremtal count of the bytes transferred so far, and is included in EVERY packet as a running total. So i guess thats how they do it, and thats what you likely need to report on. But that only exists in TCP (UDP for instance does not have this).
hope this helps.
As for me, i will likely have time to mess around with this stuff on the wekeend.
Ever grateful for your assistance here @DavidHourani
Hi all, i havent had time to look at this further. My splunk is still ingesting loads of netflow, but i havent started dev on the SPL. Seems lots of people looking for this. @DavidHourani has specifically asked for a new question to be asked on a new post, not quite sure why, its still the same dev problem we need solved, but regardless happy to follow the new thread, just pls link us in here @akg2019 so we know where to follow. Thanks guys!
Hi @DavidHourani - really appreciate your assistance here.... attached is a screenshot of some sample data thats as good as any other. Let me know if you need an actual export.
Basically (if you didnt know about netflow) the router sends periodical "flow records" back to a reciever - in this case splunk (FYI - each data packet can contain many flow records, and splunk pulls them out as an event per record).... so its a snapshot into what the routers session table is at that moment, inclusive of byte count for those transiting sessions. So if you have a long running TCP session to a DB server for instance, at minute one, it will have a byte count (bytes_in/out) of say 100.... check back 1 minute later, it migth have a byte count of say 1000, indicating 900 more bytes in that last minute.
I think the search logic needs to
- group like flows by TCP/UDP sessions which is (src_ip + dst_ip + src_port + dest_port).....
- graphing bytes over time.
- And then grabbing only say the top 10 flows by byte count.
Check the original post for the kind of flow data visualisation over time we are hoping for.
Hi David and Keiran,
I have created a new post. Please follow the below link.
Subject : How to create network monitoring report for netflow and sflow data ?
same situation for me!
i ingested netflow from our cisco-routers to splunk via Splunk app for stream.
Now i want to visualize it.
@keiran_harris do you have some results jet?
Regards, Tobias,Hi together,
same situation for me!
We also ingested netflow from our cisco-router and want to visualize it now.
@keiran_harris do you have some results jet?
I have ingested netflow and sflow wire data from our Juniper switches. But there is no visualization app with inbuilt/default dashboards. Can someone help with SPL queries or apps that can visualize the data similar to Manage Engine/Solarwinds dashboards?
I am trying to create custom dashboard report that lists the top N source to destination conversation by bit rate (bps) and traffic volume (Total MB/GB).
Post this i wanted to include other fields like port and Interface ID's as well.
Thanks for the search query. However bps is not captured directly. For example in sflow data there is no field such as bps. It has to be calculated manually. Same applies to netflow as well.
I am looking for the search queries that calculates bitrate (bps) and traffic volume (bytes transferred in MB/GB). The search query should calculate these metrics for both netflow and sflow data which has the relevant data in different field names.
Basically i am looking for network monitoring report via Splunk. Any help on this is highly appreciated.