All Apps and Add-ons

Citrix Netscaler - Appflow - Template not known (yet)

BenjaminWyatt
Communicator

I just installed the components needed to use the Splunk App for Citrix Netscaler with AppFlow. The ns_log sourcetype shows up just fine, but for the appflow sourcetype, all I see are logs with this message:

TimeStamp="2014-07-16T21:00:04"; Template="264"; Observer="1"; Address="10.2.41.254"; Port="2203"; ParseError="Template not known (yet).";

Does anybody know what this means? I can't seem to find information on it in the docs...

1 Solution

jbennett_splunk
Splunk Employee
Splunk Employee

Yeah, it means ... the templates haven't been received yet. 😉

AppFlow is IPFIX, it's a binary protocol ... in order to unpack the data you need the template definitions. IPFIX producers using UDP send the templates periodically (with Netscaler it's somewhere between every minute and once an hour ... depending on configuration), and the binary data streams can't be unpacked until the templates are received, so this message basically just lets you know that data is arriving, but the templates have not been received yet, so it can't parse the data.

View solution in original post

postmasterhg
New Member

Hi Millern4

thanks for the input. I do already run the TA_IPFIX 5.0.2 already ! unless you were talking about an additional patch on it !!

best regards

0 Karma

millern4
Communicator

Hi postmasterhg,

My issues with the template not being received and actually populating the dashboards within the NetScaler App where ultimately related to a bug within the ipfix TA provided by Splunk after a support ticket was opened.

Our issue was the templates were being consistently delivered by the NetScaler to our Heavy Forwarder, but at some point (usually within about 10 minutes) data would stop being collected and we'd see the CPU spike on the server.

Since filing the bug report, Splunk has released an updated TA for IPFIX which has resolved these issues. If you haven't tried it already I'd give it a shot and let us know if it helps.

https://apps.splunk.com/app/1801/

Thanks!

0 Karma

tred23
Path Finder

I have the lastest verison of the app however I still get the log message that the version of the .iespec file is from an older version of firmware than we have loaded.

0 Karma

postmasterhg
New Member

Hi.

thanks a lot for this in-depth explanation. I'm also facing the same issue "Citrix Netscaler - Appflow - Template not known (yet)" for several days now. It seems that the template I'm supposed to receive from the Netscaler unit onto My splunk server does not come in or is not properly function.

such a "template" is supposed to be send every 5 min, based on my netscaler unit:

> show appflow param
        AppFlow parameters

        IPFIX template refresh interval: 600 seconds
        Appname refresh interval: 600 seconds
        IPFIX flow record export interval: 600 seconds
        IPFIX UDP Path MTU: 1472 bytes
        HTTP URL logging: ENABLED
        AAA username logging: ENABLED
        HTTP cookie logging: ENABLED
        HTTP referer logging: ENABLED
        HTTP method logging: ENABLED
        HTTP host logging: ENABLED
        HTTP user-agent logging: ENABLED
        HTTP Content-Type header logging: ENABLED
        HTTP Authorization header logging: ENABLED
        HTTP Via header logging: ENABLED
        HTTP X-Forwarded-For header logging: ENABLED
        HTTP Location header logging: ENABLED
        HTTP Setcookie header logging: ENABLED
        HTTP Setcookie2 header logging: ENABLED
        HTTP Domain Name logging: ENABLED
        Log only client-side traffic: YES
        Connection Chaining: ENABLED
        Skip Cache Redirection HTTP Transaction: ENABLED
 Done

Looking at a few stats from my Netscaler units, I can verify that I'm successfully sending appflow data to splunk (setting has been done according to this set up video) and I can see those in ma splunk search requests ! :

> show appflow policy
1)      Name: AppFlow_Policy_for_Splunk
        Hits: 11191
        Undef Hits: 0
        Active: Yes

 Done
> show appflow action

1)      Name: AppFlow_Action_for_Splunk
        Collectors: AppFlow_Collector_for_Splunk
        Client-side Measurements: ENABLED
        Hits: 11191
        Action Reference Count: 1
 Done
> show appflow collector
1)      Name: AppFlow_Collector_for_Splunk
        IPv4 address: xx.xx.xx.xx
        UDP port: 4739
        Netprofile:
 Done

Is there any other way to check such a famous template reception ? Force it ??

Isn't such a kind of template, a file with iespec extension ? such a files are included in Splunk_TA_ipfix & SplunkforCitrixNetScaler apps package structure:

Splunk_TA_ipfix/default/information-elements/netscaler_10.1.iespec
SplunkforCitrixNetScaler/default/information-elements/netscaler_10.1.iespec

Couldn't we use those as a starter ?? and how ?

regards

Franck

0 Karma

jbennett_splunk
Splunk Employee
Splunk Employee

The templates are never stored on disk by this collector.

To be clear: Appflow is a just a branding thing, as far as I can tell. It really is pure IPFIX, it doesn't extend or break the RFC at all. As far as I can tell, they only call it Appflow because a certain other company calls their implementation (which predates the standardization) Netflow...

For what it's worth, I think of IPFIX as a better syslog. One with binary data transmission, and the notion of strongly typed records (template) which are like tables in a database (or sheets in excel, whatever helps you get your head around it). The idea is that the sender defines records (like a database table, or a spreadsheet) and then defines the fields (columns) for each record. The only catch is that there are many "standard" information elements (data types, if you will), but enterprises are allowed to extend that by defining their own.

The one further caveat is that RFC 5610 which shows how to express the enterprise information elements hasn't been implemented by any senders that I've encountered. We've been communicated that sort of information out-of-band. Citrix, for example, provides that information on their Appflow website (which is why you have the .iespec files in this collector).

Anyway.

According to the RFCs (of which there are many), the sender defines the templates and sends them when you first connect. That assumes a streaming protocol, so when sending over UDP, you must re-send the templates on an interval. Furthermore, the sender can (hypothetically) revoke a template and send out a new template using the same ID at any time, so unless you've received the templates after you started listening, you can't assume they're the same as they were before -- that's why we don't store them in the indexer.

In the real world, I suspect we could write out the templates and reuse them until you upgraded the Netscaler ... but that wouldn't be spec compliant 😉

millern4
Communicator

jbennett - Thanks for taking the time to explain this in further detail. This was an perfect explanation that I'm going to share internally with my team. Thanks!

jbennett_splunk
Splunk Employee
Splunk Employee

Yeah, it means ... the templates haven't been received yet. 😉

AppFlow is IPFIX, it's a binary protocol ... in order to unpack the data you need the template definitions. IPFIX producers using UDP send the templates periodically (with Netscaler it's somewhere between every minute and once an hour ... depending on configuration), and the binary data streams can't be unpacked until the templates are received, so this message basically just lets you know that data is arriving, but the templates have not been received yet, so it can't parse the data.

millern4
Communicator

Trying to better understand how the TA IPFIX is actually working here with the AppFlow data being received from the NetScalers (in my case via a Heavy Forwarder).

So from a high level perspective RFC 5101 is for the IPFIX protocol developed by Cisco which specifies templates are always contained in the flow of the data - AppFlow created by Citrix basically built their own protocol based on the original RFC and IPFIX standard if I'm interpreting this correctly.

So would the templates would ever be stored in our Indexers? Or should I expect them to always arrive with the AppFlow NetScaler data? Does this explain the reason for the TA IPFIX that contains the iespec netscaler_10.1 so that Splunk can understand how to actually decode the binary data?

Overall just trying to get a better handle on how this all works, appreciate the help.

0 Karma

BenjaminWyatt
Communicator

Perfect, the problem fixed itself. Thanks!

0 Karma

BenjaminWyatt
Communicator

Also, we are currently using v10.1 of the Netscaler.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...