topic Re: Using Splunk Stream for Netflow- now, ingesting but how to graph? in All Apps and Add-ons

Using Splunk Stream for Netflow- now, ingesting but how to graph?

keiran_harris — Sun, 10 Jun 2018 22:51:52 GMT

Hi splunk gurus!

Long weekend here in Australia and i thought id finally get around to ticking something off my wish list: netflow my home network.

So ive got a cisco adsl router thats successfully streaming netflow to my splunk box (verified first with tcpdump). At the splunk side, i started off down one path (“Netflow Analytics” until i realised you had to pay, a lot, for that!)... then some searching in here pointed me to “splunk stream”, which seems robust, is free, now installed, and happily gobbling up my netflow stream! See attached photo.

Which brings me to the fun part (and my question). Where can i find some pre-canned SPL to start plotting my traffic on pretty graphs? The Stream UI doesnt look to be setup for this. I know i could start to write myself but its a relatively complex dataset, and surely this has been done lots before, so i shouldnt have to reinvent the wheel. So if anyone can point me at some SPL (or an app!) that would be great!

Thanks in advance all.
Keiran.

PS- this is the sort of graph I'm hoping to create (from the paid app - https://splunkbase.splunk.com/app/489):

Re: Using Splunk Stream for Netflow- now, ingesting but how to graph?

keiran_harris — Fri, 15 Jun 2018 12:38:23 GMT

Giving this a nudge so it bubbles up again for some viewers who can help!

Re: Using Splunk Stream for Netflow- now, ingesting but how to graph?

xiongwei002 — Thu, 23 Aug 2018 05:18:44 GMT

Agree, top up, I need it too.

Re: Using Splunk Stream for Netflow- now, ingesting but how to graph?

Zolrak — Sat, 12 Jan 2019 12:27:24 GMT

Looking for some light into this too, I'm under the same boat, I loved the Netflow pay app and it works but will like to have the same type of dashboard with the Stream app.

Re: Using Splunk Stream for Netflow- now, ingesting but how to graph?

tobiasgoevert — Fri, 22 Feb 2019 06:34:36 GMT

Hi Together,

same situation for me!
i ingested netflow from our cisco-routers to splunk via Splunk app for stream.
Now i want to visualize it.

@keiran_harris do you have some results jet?

Regards, Tobias,Hi together,

same situation for me!
We also ingested netflow from our cisco-router and want to visualize it now.
@keiran_harris do you have some results jet?

Regards, Tobias

Re: Using Splunk Stream for Netflow- now, ingesting but how to graph?

akg2019 — Mon, 20 May 2019 06:24:44 GMT

Hi,

I have ingested netflow and sflow wire data from our Juniper switches. But there is no visualization app with inbuilt/default dashboards. Can someone help with SPL queries or apps that can visualize the data similar to Manage Engine/Solarwinds dashboards?

Thanks,
AKG

Re: Using Splunk Stream for Netflow- now, ingesting but how to graph?

DavidHourani — Mon, 20 May 2019 06:31:54 GMT

sure, what are you trying to build ?

Re: Using Splunk Stream for Netflow- now, ingesting but how to graph?

akg2019 — Mon, 20 May 2019 06:47:20 GMT

Hi David,

I am trying to create custom dashboard report that lists the top N source to destination conversation by bit rate (bps) and traffic volume (Total MB/GB).

Post this i wanted to include other fields like port and Interface ID's as well.

Thanks,
AKG

Re: Using Splunk Stream for Netflow- now, ingesting but how to graph?

DavidHourani — Mon, 20 May 2019 06:53:54 GMT

@akg2019, you're looking for something like this :

index=whereYourDataIs sourcetype=yourSourcetype | stats avg(bps) as bitRate, sum(bps) as volume by src dest

Re: Using Splunk Stream for Netflow- now, ingesting but how to graph?

akg2019 — Mon, 20 May 2019 07:06:19 GMT

Hi David,
Thanks for the search query. However bps is not captured directly. For example in sflow data there is no field such as bps. It has to be calculated manually. Same applies to netflow as well.

I am looking for the search queries that calculates bitrate (bps) and traffic volume (bytes transferred in MB/GB). The search query should calculate these metrics for both netflow and sflow data which has the relevant data in different field names.

Basically i am looking for network monitoring report via Splunk. Any help on this is highly appreciated.

Re: Using Splunk Stream for Netflow- now, ingesting but how to graph?

DavidHourani — Mon, 20 May 2019 07:10:08 GMT

Can you make it into a new question please and include a sample event line ? We can work from there

Re: Using Splunk Stream for Netflow- now, ingesting but how to graph?

akg2019 — Mon, 20 May 2019 07:45:36 GMT

Sure David

Re: Using Splunk Stream for Netflow- now, ingesting but how to graph?

keiran_harris — Mon, 20 May 2019 08:02:57 GMT

Hi all, i havent had time to look at this further. My splunk is still ingesting loads of netflow, but i havent started dev on the SPL. Seems lots of people looking for this. @DavidHourani has specifically asked for a new question to be asked on a new post, not quite sure why, its still the same dev problem we need solved, but regardless happy to follow the new thread, just pls link us in here @akg2019 so we know where to follow. Thanks guys!

Re: Using Splunk Stream for Netflow- now, ingesting but how to graph?

akg2019 — Mon, 20 May 2019 08:11:56 GMT

Hi David and Keiran,

I have created a new post. Please follow the below link.

https://answers.splunk.com/answers/747044/how-to-create-network-monitoring-report-for-netflo.html?minQuestionBodyLength=80

Subject : How to create network monitoring report for netflow and sflow data ?

Thanks,
AKG

Re: Using Splunk Stream for Netflow- now, ingesting but how to graph?

DavidHourani — Mon, 20 May 2019 08:27:14 GMT

link is not working for me XD

Re: Using Splunk Stream for Netflow- now, ingesting but how to graph?

akg2019 — Mon, 20 May 2019 08:46:11 GMT

Hi David,

The new post status is "This post is currently awaiting moderation. If you believe this to be in error, contact a system administrator."

Not sure when it will get approved.

Re: Using Splunk Stream for Netflow- now, ingesting but how to graph?

DavidHourani — Mon, 20 May 2019 08:50:47 GMT

ouch... okay, let me know when it's up, and if you want go ahead and share some sample (anonymized) logs here so we can work with it.

Re: Using Splunk Stream for Netflow- now, ingesting but how to graph?

keiran_harris — Wed, 30 Sep 2020 00:38:01 GMT

Hi @DavidHourani - really appreciate your assistance here.... attached is a screenshot of some sample data thats as good as any other. Let me know if you need an actual export.

Basically (if you didnt know about netflow) the router sends periodical "flow records" back to a reciever - in this case splunk (FYI - each data packet can contain many flow records, and splunk pulls them out as an event per record).... so its a snapshot into what the routers session table is at that moment, inclusive of byte count for those transiting sessions. So if you have a long running TCP session to a DB server for instance, at minute one, it will have a byte count (bytes_in/out) of say 100.... check back 1 minute later, it migth have a byte count of say 1000, indicating 900 more bytes in that last minute.

I think the search logic needs to
- group like flows by TCP/UDP sessions which is (src_ip + dst_ip + src_port + dest_port).....
- graphing bytes over time.
- And then grabbing only say the top 10 flows by byte count.
Check the original post for the kind of flow data visualisation over time we are hoping for.

Re: Using Splunk Stream for Netflow- now, ingesting but how to graph?

akg2019 — Wed, 30 Sep 2020 00:36:44 GMT

Hi David,

The new post is yet to be approved. Lets continue working in this thread.

Basically i am looking for network monitoring report via Splunk similar to Manage Engine/Solarwinds/Ipswitch dashboards.

In the report i wanted to calculate metrics such as bitrate (bps) and traffic volume (bytes transferred in MB/GB).
The search query should calculate these metrics for both netflow and sflow data which has the relevant data in different field names.

Sample ingested sflow V5 and netflow V9 data fields are attached.

Can you please help in creating a standard network monitoring report that contains source_IP , dest_IP , Port , Bitrate (bps) , Bytes (MB/GB) etc.. for a given time range.

Thanks,
AKG

Re: Using Splunk Stream for Netflow- now, ingesting but how to graph?

DavidHourani — Mon, 20 May 2019 11:08:47 GMT

so if my understanding is correct, you have the source and dest port in sflow, the bytes_in in netflow and you wish to combine both of them ?

In that case running a search like this should do the trick :

index=yourIndex sourcetype=yoursourcetype | stats values(bytes_in) as bytes_in, values(dest_port) as dest_port values(src_port) as src_port by src_ip, dest_ip | eventstats sum(bytes_in) as volume, avg(bytes_in) as Bps by src_ip, dest_ip