Splunk Stream - Netflow v9 - Certain fields not really show up

[jvanhambelgium]

Hi,

I've got the latest Splunk Stream app installed & configured to accept Netflow v9 events from my router. This part works fine actually.
However, when starting to dig deeper in "useful" fields it seems I'm missing a few...I would expect the Stream App to be able to cope with everything "standard" within v9/IPFIX packets/templates.

When going to the STM App -> Configuration -> Configure Stream -> Netflow -> "Edit" you get this nice list of about 156 fields which I all enabled.

Now I've taken Wireshark capture from the v9 data arriving at my Splunk server and the "template" contains these fields below.

Field (20/23): postNATSourceIPv4Address
Type: postNATSourceIPv4Address (225)
Length: 4

Field (21/23): postNATDestinationIPv4Address
Type: postNATDestinationIPv4Address (226)
Length: 4

Field (22/23): postNAPTSourceTransportPort
Type: postNAPTSourceTransportPort (227)
Length: 2

Field (23/23): postNAPTDestinationTransportPort
Type: postNAPTDestinationTransportPort (228)
Length: 2

And a typical populated capture would look like this :

Cisco NetFlow/IPFIX
Version: 9
Count: 7
SysUptime: 873590.040000000 seconds
Timestamp: Jun 22, 2020 10:03:08.000000000 CEST
CurrentSecs: 1592812988
FlowSequence: 22
SourceId: 0
FlowSet 1 [id=256] (7 flows)
FlowSet Id: (Data) (256)
FlowSet Length: 532
[Template Frame: 1]
Flow 1
[Duration: 0.000000000 seconds (switched)]
StartTime: 873528.130000000 seconds
EndTime: 873528.130000000 seconds
Packets: 1
Octets: 86
InputInt: 15
OutputInt: 14
SrcAddr: IP.OF.INTERNAL.PC
DstAddr: SOME.PUBLIC.ISP.DNSADDRESS
Protocol: UDP (17)
IP ToS: 0x00
SrcPort: 51020 (51020)
DstPort: 53 (53)
NextHop: SOME.PUBLIC.ISP.DNSADDRESS
DstMask: 0
SrcMask: 0
TCP Flags: 0x00
Destination Mac Address: Router12_12:12:c6 (61:3c:61:31:11:b1)
Source Mac Address: ASRockIn_84:01:36 (d0:50:99:84:01:36)
Post Source Mac Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
Post NAT Source IPv4 Address: MY.PUBLIC.ISP.ADDRESS
Post NAT Destination IPv4 Address: SOME.PUBLIC.ISP.DNSADDRESS
Post NAPT Source Transport Port: 51020
Post NAPT Destination Transport Port: 53

Looking again at my Splunk Enterprise intstallation, there is this "vocabulaire" file under

opt/splunk/etc/apps/Splunk_TA_stream/default/vocabularies

It contains the "netflow.xml" file with (what I think) all field that can be "interpreted" / "decoded" as the Netflow packets arrive.

<Term id="netflow.postNATSourceIPAddress">
<Term id="netflow.postNATDestinationIPAddress">
<Term id="netflow.postNAPTSourceTransportPort">
<Term id="netflow.postNAPTDestinationTransportPort">

So ... these 4 fields seem to already be part of the default vocabulary ... yet they never show up as any accessible "field" in Splunk ?

In a moment of madness, I've edited the file below and make some additions.

(router = Mikrotik = IANA Vendor ID 14988)

user@splunky:/opt/splunk/etc/apps/Splunk_TA_stream/default# more streamfwd.conf
[streamfwd]
port = 8889
ipAddr = 127.0.0.1

netflowReceiver.0.ip = IP.OF.MY.SPLUNK
netflowReceiver.0.port = 9995
netflowReceiver.0.decoder = netflow

netflowElement.0.enterpriseid = 14988
netflowElement.0.id = 225
netflowElement.0.termid = netflow.postNATSourceIPAddress

netflowElement.1.enterpriseid = 14988
netflowElement.1.id = 226
netflowElement.1.termid = netflow.postNATDestinationIPAddress

netflowElement.2.enterpriseid = 14988
netflowElement.2.id = 227
netflowElement.2.termid = netflow.postNAPTSourceTransportPort

netflowElement.3.enterpriseid = 14988
netflowElement.3.id = 228
netflowElement.3.termid = netflow.postNAPTDestinationTransportPort

...then stop/start my Splunk but these fields don't show up with the 156 possible fields under the "stream config" tab (see earlier)

To cut a long story short : Where are these fields ? Why are they not showing up since they are hitting the Splunk Stream App and it seems they are "known"

Thanks!

[thobreit]

Hi,

I had the same challenge a few weeks ago and was able to make it work with the latest Splunk Stream version 8.1.

This version supports custom apps to use propietary netflow configurations. Link to Splunk Documentation 

You can reuse the existing definitions in the stream app from default/streams & default/vocabularies and copy them into your custom app.  If you want to use all existing netflow fields you have to keep in mind netflow utilizes additional vocabularies like ip.xml or flow.xml.

In the ipfixmap.conf you can add the NAT fields with the netflow id. I used this table to get the correct values.

Fields you don't define in ipfixmap.conf will use their default field id from the splunk stream app.

When configured correctly you can choose you new custom stream in the Splunk Stream WebGUI.

The ipfixmap.conf i used to get the NAT fields:

netflowElement.0.id = 225
netflowElement.0.termid = netflow.NATSourceIPAddress

netflowElement.1.id = 226
netflowElement.1.termid = netflow.NATDestinationIPAddress

netflowElement.2.id = 227
netflowElement.2.termid = netflow.NAPTSourceTransportPort

netflowElement.3.id = 228
netflowElement.3.termid = netflow.NAPTDestinationTransportPort

Regards

Thomas

[diptij]

Did you figure this out?  if so, what did you do?

[jvanhambelgium]

Hi,

Actually never managed to get it to work.

I must also admit the v9/IPFIX Netflow on the Mikrotik RouterOS is not without its problems either...

So for the moment I've settled for Netflow v5 which at least gives me some output. This works fine.