Splunk Stream - Cisco IpFIX . Can't figure out how to ingest IPFIX data for application URL/URI/source/dest

[sudoritz]

Im trying to find some documentation to help aid in ingesting Custom IPfix outside 1-400 IDs. But I read that there's not much documentation in this arena hehe .. Here's what I have tried:

Main goal is to pretty much ingest IPFIX data for application URL/URI / source/dest other Netflow stats but it seems i need to code either in vocabularies or something else.

Cisco ASR 1004 --> streamfwd standalone app --> SH / indexer load

I've noticed Template ID of 294 and enterprise ID of 9

but i dont see it in ipfix.xml in the IETF org assignments

connection client ipv4 address ID = 12236
connection server ipv4 address ID = 12237

i tried setting this in streamfwd.conf

cat streamfwd.conf

[streamfwd]
    port = 8889
    netflowReceiver.0.ip = 10.1.1.1
    netflowReceiver.0.protocol = udp
    netflowReceiver.0.port = 9991
    netflowReceiver.0.decoder = netflow
    netflowReceiver.0.decodingThreads = 8

    netflowElement.0.enterpriseid = 9
    netflowElement.0.id = 12235
    netflowElement.0.termid = cisco.12235
    netflowElement.1.enterpriseid = 9
    netflowElement.1.id = 12236
    netflowElement.1.termid = cisco.12236

and tried setting this in vocabularies

vocabularies]# cat cisco.xml

<?xml version="1.0" encoding="UTF-8"?>
<CmConfig xmlns="http://purl.org/cloudmeter/config" version="7.1.1">
  <Vocabulary id="cisco">
    <Locked>true</Locked>
    <Name>Cisco Netflow Protocol Vocabulary</Name>
     <Term id="cisco.12235">
        <Type>blob</Type>
        <Comment>12235 status.</Comment>
      </Term>
<Term id="cisco.12236">
      <Type>blob</Type>
      <Comment>12236 status</Comment>
</Term>
   </Vocabulary>
</CmConfig>

theres some things im trying to figure out and stitch together like how do i know how to state its a uint32/64

i tried to look at the exporter part of the router to then build in the vocabularies
size1=unsigned8
size4=unsigned32
size8=unsigned64
size32=string
size40=string

but its not 1-1 on some of them so im kinda lost on how i can bridge some of these inbound.

This is the Exporter information from our cisco router showing Template ID of 294 along with IDs and Ent ID

Client: Flow Monitor cisco-flow
  Exporter Format: IPFIX (Version 10)
  Template ID    : 294
  Source ID      : 1280
  Record Size    : 95 + var
  Template layout
  _____________________________________________________________________________
  |                 Field                   |    ID | Ent.ID | Offset |  Size |
  -----------------------------------------------------------------------------
  | connection client ipv4 address          | 12236 |      9 |      0 |     4 |
  | connection server ipv4 address          | 12237 |      9 |      4 |     4 |
  | ip dscp                                 |   195 |        |      8 |     1 |
  | ip protocol                             |     4 |        |      9 |     1 |
  | connection client transport port        | 12240 |      9 |     10 |     2 |
  | connection server transport port        | 12241 |      9 |     12 |     2 |
  | routing vrf input                       |   234 |        |     14 |     4 |
  | connection initiator                    |   239 |        |     18 |     1 |
  | connection id                           | 12242 |      9 |     19 |     4 |
  | flow observation point                  |   138 |        |     23 |     8 |
  | application id                          |    95 |        |     31 |     4 |
  | flow direction                          |    61 |        |     35 |     1 |
  | flow sampler                            |    48 |        |     36 |     1 |
  | services waas segment                   |  9252 |      9 |     37 |     1 |
  | services waas passthrough-reason        |  9253 |      9 |     38 |     1 |
  | application http uri statistics         |  9357 |      9 |     39 |   var |
  | application http host                   | 12235 |      9 |     41 |   var |

*** i have it coming in Splunk cause i edited the app /streams/netflow and i see 12235 but it doesnt show like its correct) mabye because i did it with blob.***

Does anyone have example custom Cisco setup (i thought this would be like an easy 1-1

also in my streamfwd log i have this but not sure if i built it right

2017-10-07 11:06:29 WARN 140323433604864 stream.NetflowReceiver - NetFlowDecoder::decodeFlow Unable to decode flow set data. No template with id 261 received for observation domain id 6 from device 10.0.1.1 . Dropping flow data set of size 1358

[sudoritz]

oops i need to move my answer to my comment when approved

[sudoritz]

question1

how do i know from ciscos SIZE output based on the field how to map out

size1=uint8
size2=uint16
size4=uint32
size8= uint64
size32=string
size40=string

The Type tag must consist of one of the following supported data types:
uint8, uint16, uint32, uint64, shortstring, string, longstring, blob.

But not sure how to get size32/40 on shortstring/longstring/blob

i cant find in Cisco where it states on how to on those.

[alacercogitatus]

Hey @sudoritz!!!

Read here: https://www.aplura.com/splunk/extending-splunk-stream-vocabularies-using-ipfix/

If you have further questions, comment below, or hop on slack, find the #stream channel, and ping me!!!

Thanks!

[lfedak_splunk]

Here's the link to Slack! http://splk.it/slack